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PREFACE 


The impetus for this study derived mainly from two factors. 
First the intelligence community had never taken a comprehensive 
look at the role of national intelligence estimates in the 
decision-making processes of government. Secondly, estimates 
have been increasingly subject to criticism in recent years, 
and a serious effort to evaluate them seemed in order. Con- 
sequently, a study team composed of experienced Agency officers 
was assembled and work begun in July 1976, under the auspices 
of CIA’s Center for the Study of Intelligence. 

The intelligence community produces a wide variety of 
products which make judgments about the future, ranging from 
crisis reports and short-term analytical articles through 
National Intelligence Estimates (NIEs) and similar interagency 
products. The study deals only with the last of these, i.e., 
interag ency papers produced by National Intelligence Officers 


The team was composed of 


member of the former 


Board of National Estimates and a National Intelligence Officer 
fRet. 1 : T I of the former Office of Current Intelligence, 

Directorate of Intelligence, and presently Director of t he Center 
for the Study of Intelligence in the Office of Training; [ 


of the Directorate of Operations, who served with the Office 


of National Estimates in its early years and recen tly comple ted 
a tour on the National Security Council Staff; and I I of 

the former Office of Current Intelligence, now assigned to the 
Current Reporting Group, Directorate of Intelligence. 
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(NIOs) on behalf of the Director of Central Intelligence (DCI) 
(although some findings of the study may be germane to other 
products). Included are NIEs and Special NIEs, now produced in 
considerably fewer numbers than in former years. Also included 
are National Intelligence Analytical Memoranda (NIAM) and 
Interagency Intelligence Memoranda (IIM), which have existed 
only since 1973. The NIAM, infrequently used, is lengthier 
and more detailed than the typical NIE. The IIM is produced 
under the aegis of the DCI, but is not passed upon by the 
National Foreign Intelligence Board (NFIB) and is a less 
formal and more flexible publication than the NIE. 

The study methodology adopted placed considerable emphasis 

on conducting interviews with a wide range of users and pro- 

2 

ducers of estimates. Valuable supplemental and background 
information was acquired from existing classified and unclas- 
sified literature about estimates and from published estimates. 
Devising a methodology for evaluating estimates themselves 
proved impracticable, for the reasons given in Appendix A. 
Appendix A also contains a fuller explanation of the method- 
ology used in the study, as well as information on the 66 
interviews conducted by the study team. A selective bib- 
liography is included as Appendix B. 

2 * 

It is important to remember that the interviews were con- 
ducted prior to the change in Administration which occurred 
in January 1977 and that the comments apply to the intelli- 
gence and policy processes which existed at that time. 
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It did not prove easy to generalize about the responses 
of those interviewed. Time did not permit us to cover all the 
issues with each of the individuals interviewed. The responses 
covered a wide spectrum, and the respondents did not fall 
neatly into categories; typically, reactions to estimates 
were mixed. It was also necessary to allow for personal dif- 
ferences, e.g., essentially the same view might be expressed 
by one person in low key and by another with vigor and passion, 
or differences in view might be disguised by subtleties of 
language. Further, a study of this kind tends to invite a 
critical approach, as does the asking of questions in an inter- 
view situation. Efforts were made to guard against these and 
similar hazards, and to concentrate in the report on certain 
recurring themes which rather clearly emerged from the interviews. 

The report covers three broad topics: the product and 
its audience, the process of producing estimates, and the 
relationship between producer and user. The first topic is 
considered in Chapter II, which sets forth the views of users 
concerning the quality and utility of estimates, and con- 
trasts these views with the traditional perception of the 
role and function of estimates. This chapter takes up two 
other matters: what estimates should attempt to do for the 
reader, and for whom they should be prepared. The estimative 
process is discussed in Chapters III and IV. Chapter III 
discusses certain specific aspects of the production process, 
such as methods of presentation and the coordination process. 

iii 
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Chapter IV covers the question of the organization for pro- 
ducing estimates. The relationship between producers and 
users is taken up in Chapter V, which discusses, inter alia , 
the benefits and risks of close association, and the means 
by which the needs of users can be conveyed to producers of 
estimates. The final chapter has some thoughts on the future 
of estimates. 

Gratitude is tendered to the many people who agreed to 
be interviewed and who so generously contributed time and 
thought to this study. It could not have been prepared with- 
out them. 
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SUMMARY 

I. SETTING THE STAGE (pp . 19-22) 

National Intelligence Estimates (NIEs) have changed over 
the years from short, narrowly focused papers dealing with 
near-term problems to include more comprehensive, analytical 
studies of longer-range issues. This change resulted from: 

--greater analytical capabilities; 

--an enlarged data base; and, 

--changing requirements, as readers grew more 

numerous, more sophisticated, and more demanding. 

The role and importance of estimates in policy making has 
varied with Administrations, particularly in response to: 

--the structure of the policy making machinery 
and the place accorded estimates; 

--the attitude of top policy makers toward intel- 
ligence; and, 

--the quality and relevance of estimates as 
perceived by the principal users. 

Reaching their zenith in the early 1960's, estimates sub- 
sequently declined in prestige and drew increasingly sharp 
criticism. The criticism contributed to the decision of the 
Director of Central Intelligence (DCI) in 1973 to abolish the 
Board and Office of National Estimates (ONE) and entrust the 
preparation of estimates to a group of National Intelligence 
Officers (NIOs) , 
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II. ESTIMATES AS AN INSTRUMENT OF POLICY SUPPORT (pp . 23-50) 

The criticism of estimates has continued since the change, 
however, and this chapter sets forth the views of policy makers 
and other consumers on the present quality and utility of esti- 
mates and comments on how and for whom they should be written. 

The Traditional Doctrine (pp . 23-24) 

National estimates were intended at their outset to be the 
most authoritative appraisals available to the top levels of 
government on foreign developments of national security concern. 
Issued by the DCI, they were to be forward looking and predictive 
rather than historical and descriptive, of high quality and 
objectivity, and national products with respect to subject 
matter, audience, and process of production. The interviews 
for this study were conducted with this doctrine in mind and 
the results were measured against it. 

How Estimates Fared (pp . 25-34) 

Against the traditional standard, estimates did not fare 
well. Although highly praised by some users, and found use- 
ful in one way or another by most others, they were judged 
in the aggregate to fall well short of the traditional ideal. 

They clearly have not played the important role envisioned 
for them in the national security decision process. 

The Negative Side (pp . 25-28). Estimates seldom reach 
the top levels of their intended audience- -the President and 
members of the National Security Council (NSC) . Such of their 
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content as does is usually included in memoranda or briefings 
prepared at lower levels, where estimates are widely received, 
and read to varying degrees. The complaints about estimates 
focused on quality and relevance; estimates were criticized 
for being: 

--irrelevant to, or oblivious of, the specific 
policy problems of the readers; 

--insufficiently analytical and overly descriptive; 

--conservative and imprecise in their judgments 
about the future; 

--inadequate in explaining judgments and con- 
clusions, and in discussing the alternatives 
considered and discarded; and 5 

--unable to contribute much that is unique or 
not already known to the policy maker, par- 
ticularly on political subjects. 

T he Positive Side (pp. 28-30), Some respondents were high 
in their praise of estimates and others were on the whole well 
satisfied with them; almost all found them useful in one way or 
another. For example: 

--those with strongly positive views, including 
two former cabinet members, tended to be less 
concerned about the relevance of estimates to 
immediate policy issues, and valued them for 
their presentation of a disinterested view; 

--others with a positive view expected less of 
estimates, and were not troubled by the 
deficiencies perceived by the more critical; 

--most users valued estimates for their balance 
and professionalism, because they pulled 
together all that was known about a subject, 
and because they helped assure the reader 
that he had considered all the factors 
bearing on a problem. 
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Other Findings (pp. 30-34). Estimates got good marks 
for ob j ectivity- - some users considered this one of their 
principal virtues. Most users who said they had observed 
bias, considered it a comparatively minor problem, easily 
discerned and adjusted to, and the consequence of human 
imperfection rather than of deliberate intent. 

It proved difficult to get the views of users on dif- 
ferences in the quality and utility of estimates since the 
1973 change in the production system, because of the turnover 
of officers in policy positions. There was, however, a fair 
degree of consensus between users and producers on two points 

--the NIO system has produced a modest improve- 
ment in the relevance of estimates and some- 
what greater improvement in, the responsiveness 
of the system; but, 

--estimates are more uneven in quality than those 
produced before 1973, because of the new drafting 
procedures . 

The interviews revealed different reactions to different 
kinds of estimates: 

--those on military, scientific, technical and 
economic subjects were better received than 
those on political subjects, not because of 
differences in quality, but because most 
users were less able to handle the complex 
data, perform their own analysis, and reach 
their own conclusions. 

We found little or no support for criticism heard in 
recent years concerning: 

--the proliferation of intelligence publications 
containing estimates; 
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--the issuance of departmental as well as national 
estimates; or, 

--the absence of an explicit scale of probabilities 
in estimates. 

Why the Gap? (pp . 34-40) 

The substantial disparity between traditional expectations 
and what users said is attributable in large part to deficiencies 
in the product: 

--the failure to be fully responsive to the policy 
question; 

--the failure to be sufficiently venturesome; and, 

--inadequacies in drawing implications for U.S. 
policy . 

Some Un r ealized Assumptions (pp. 35-38). There are 
other reasons for the disparity, which individually and 
collectively are of considerable importance. A basic one 
is that the traditional doctrine puts an unrealistic burden 
on the DCI and the intelligence community. It rests on some 
unstated and unrealized assumptions. 

One is that estimates would have a major influence on 
the formulation of national security policy: 

--in fact, estimates have played only a modest 
role, partly because 

--security policy is not directly driven by facts, 
analyses and resulting judgments; it is the com- 
plex product of an often lengthy and untidy pro- 
cess, in which many other considerations come 
into play. 

A second assumption is that policy makers would seek and 
welcome the contributions of estimative intelligence, even 
when they cast doubt on current policy: 

5 
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--in fact, policy makers often believe that they 
can estimate as well as the intelligence spe- 
cialist, if not better; also, 

--estimates may say unwelcome things and cast 
those who produce them in the role of 
troublemakers . 

A third assumption is that the relationship between policy 
and intelligence would be close, and communication free and 
complete : 

--this relationship has usually not existed; it 
has been very weak in recent years; moreover, 

--there is an absence of structure for system- 
atically insuring that estimates are part of 
the policy process. 

Other Reasons (pp. 38-40). Two other circumstances have 
contributed to the failure of estimates to play their prescribed 
role. One is that the foreign policy establishment tends to be 
highly operational , and to focus on the short term and highly 
specific matters immediately before it. However: 

--estimates were originally intended not merely 
to support day-to-day operations, but as con- 
tributions to the formulation of basic, long- 
term national security policy; 

--from this perspective, the difficulty may be 
with the way policy is formulated. Government 
institutions, such as State's Policy Planning 
Staff, which were designed to assist with long 
range policy, usually focus instead on short 
term issues. 

Secondly there has been suspicion and distrust of estimates 
at the top, and this has had serious effects on their use. 

--Unless estimates are welcomed and read at the 
top, they are not likely to be taken seriously 
elsewhere. 


6 


Approved For Release 2002/11/04 : CIA 




-00630A000300040001 -3 



Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 


CONFIDENTIAL 

The Mark e t for Estimates (pp. 40-50) 

Even the most critical users believe that estimates have 
a role to play in the policy process. 

What Shoul d Estimates D o? (pp. 42-46). Users believe 
that estimates should: 

--identify policy problems not yet in the fore- 
front, and pose questions about them; 

--sort out the facts in complicated situations 
where such facts are elusive, apparently con- 
tradictory or fast changing; 

--identify and evaluate the forces at work and 
their interplay, and discuss how their con- 
tinuation or manipulation could affect the 
final outcome; 

--judge the consequences of ongoing developments 
for U.S. policy; and, 

--judge foreign reactions to U.S. policies, present 
or contemplated. 

There was a relative lack of interest in specific pre- 
dictions of future events such as coups, elections, or changes 
of government. 

Most striking about these comments is their reaffirmation 
of the traditional doctrine, with its emphasis on the analysis 
of forces, trends and their implications for the U.S. in a 
context analytical and forward looking, rather than descriptive 
and current. 

For Whom Should Estimates Be Wri t ten ? (pp . 46-48). The 
nature of the audience is important, for it affects the way 
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estimates are organized, their level of detail and how they 
present facts, analyses and conclusions. 

Some of those interviewed thought that estimates should 
be written with readers at the very top in mind--the President, 
the NSC, and other cabinet members --not because estimates would 
regularly be read there, but because so aimed, they would catch 
other readers along the way. 

--But the perspectives and needs of those at 
the top are not necessarily the same as those 
of the individuals who support them. 

--Some estimates will be read at the top, par- 
ticularly those that the DCI believes should 
be read there, and which he urges upon the 
senior policy makers. 

Most respondents believed that estimates should be aimed 
at the Assistant Secretary of State, White House Staff or com- 
parable level, and we agree. 

--These are the officials who set the terms for 
the policy debate by formulating the options 
and alternatives, who enjoy the confidence of 
policy makers at the highest levels, and who 
constitute the highest level combining expertise 
and the power to act on many problems. 

How Should Estimates Be Written? (pp. 48-50)- Such 
readers are able and knowledgeable, have access to the same 
material as estimators, and feel competent to reach judgments 
on the basis of their own analysis. If estimates are to 
appeal to this audience they must: 

--emphasize analysis rather than description, 
show the relationships among data, analysis 
and conclusions, and describe the thought 
process by which the estimators came to 
their judgments; 
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--explain what issues were contentious and what 
was disgarded and why, and set forth any dif- 
ferences of opinion; 

--describe continuity and change as compared, 
with previous estimates, and identify earlier 
material now judged incorrect; and, 

--clearly state the implications of their 
analysis and conclusions for U.S. policy. 


I. IMPORTANT ELEMENTS IN THE PRODUCTION PROCESS FOR NATIONAL 
ESTIMATES (pp . 51-79) 

This chapter discusses the views of producers, some users, 
and the study team on aspects of the process by which estimates 
are produced. 

Self- In it iated Estimates (pp. 51-52) 

About half of all national estimates are initiated by the 
intelligence community and most producers and users who com- 
mented on the subject found this satisfactory. A very few 
felt strongly that, to avoid irrelevant papers, estimates 
should be produced only on request, but the majority and the 
study team believe that producers have a duty to initiate an 
estimate when they perceive a development of significance 
for U.S. policy. 

y 

Terms of Reference (pp. 52-54) 

The degree to which users participate in preparing the 
terms of reference is likely to determine the real relevance 
of an estimate to the needs and interests of its main recipi- 
ents. A formalized procedure providing for such consultation 
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should be instituted. The risk of predetermining the outcome 
by a conscious or unconscious "loading" of the questions 
appears minimal and manageable. 

Multidisciplinary Work (pp , 54 - 57 ) 

Users and producers alike agreed on the need for more 
cross-disciplinary or analytically-integrated estimates, and 
on the difficulty of doing them. A common complaint was that 
readers were left to synthesize separate sections on political, 
economic, military and other subjects. Bureaucratic separa- 
tion and increasing specialization among analysts were cited 
as the causes for this situation. There is, however, little 
agreement on how best to accomplish good multidisciplinary 
synthesis. At a minimum, it probably requires bringing 
various analysts together under an effective project leader 
for wide-ranging "synthesizing discussion" before drafting 
begins . 

New Ana l ytical Methodologies (pp- 57-59) 

Although some critics fault the estimative process for 
not incorporating more quantitative, mathematical, and 
systems-oriented methodologies, we found little support 
for this charge. There was a great deal of skepticism about 
the use of computers and other new tools, and producers saw 
some risks in the use of new methodologies. Nonetheless, it 
is important to keep up with the state of the art, and some 
new techniques appear to have at least limited applicability. 

10 
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Competin g Analysis (pp. 59-63) 

Another concept that has received attention recently 
is the creation of "competing centers of analysis." The only 
extensive effort to provide an alternative analytical approach 
was the widely-publicized and controversial "B-team" experi- 
ment. Undertaken last year during the preparation of the 
annual estimate on Soviet strategic forces, it involved a 
team of experts from outside the intelligence community. 

There are many practical problems in such an effort, however, 
and the best insurance of proper analytic competition probably 
lies in the skill, perception, objectivity, and intellectual 
rigor of the estimative manager rather than in any organiza- 
tional technique. 

Net Assessments (pp. 63-66) 

Net assessments involving the U.S. require access to 
data on U.S. forces, weapons and capabilities. Those inter- 
viewed agreed fully that the intelligence community should 
neither conduct them nor include them in estimates, because 
of the inordinate risk of transforming estimators into 
advocates or opponents of particular U.S. weapons systems 
or policies. Net assessments comparing the capabilities of 
two or more foreign countries are an acceptable and at times 
essential part of national estimates, and there is need for 
more of them. 
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Coordinatio n (pp. 66-70) 

This process brings together experts from the various 
agencies to debate and revise the text of a draft estimate. 
Coordination is central to the. concept of nationa l intelli- 
gence, allowing the DCI as the nation's chief intelligence 
officer to set forth his views while requiring other par- 
ticipants either to agree or to express and explain their 
disagreements. Substantively, coordination brings the 
talents of the entire intelligence community to bear, makes 
coverage of the pertinent issues more likely, and, at its 
best, helps to define and sharpen issues. 

The best way to avoid the pitfalls of coordination, such 
as masking divisions and fuzzing conclusions, is to select a 
chairman for coordination meetings who is tough-minded, inde- 
pendent, judicious and skilled in running a meeting, and to 
remember that consensus is often not what users want or need. 
Dissent, S ummaries and Classification (pp. 70-74) 

Consumers welcome the presentation of conflicting views 
on controversial and complex matters, and they are sufficiently 
sophisticated to distinguish between dissents reflecting sub- 
stantive differences and those taken primarily to support 
bureaucratic positions. The present trend of incorporating 
dissents in the text of estimates, rather than in footnotes, 
is clearly favored by users, some of whom want also to see 
an elaboration of the rationale behind a dissent. 

12 
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Consumers emphasized that a tightly written and accurate 
summation of the key conclusions is the best possible device 
for ensuring high-level attention to the message of an estimate. 
Many did not object to lengthy estimates so long as they led 
off with a crisp, well -written summary. The importance of a 
good summary can hardly be overemphasized. 

Policy makers strongly favored the lowest possible security 
classification for estimates as a means of widening their 
audience and enhancing their utility. Where possible, highly 
classified material seems best handled in separate, more 
restrictively distributed annexes, so that a lower classifica- 
tion can be given to the basic estimate. 

Prese ntation , "Post-Mortems" and Updates (pp. 74-79) 

While recognizing that both written papers and oral 
briefings have certain advantages, almost all consumers com- 
menting on the two methods clearly preferred to get most if 
not all estimative material in written form. The bulk of 
any effort to Improve presentational formats ought therefore 
to go into making written estimates as succinct, readable, 
and responsive to different levels of need as possible. 

Post-mortems can be useful if done sparingly, and if 
they include feedback from consumers. 

There was almost no support from consumers for a regularly 
scheduled revision and update of estimates, except for the 
annual Soviet strategic estimate, NIE 11-3/8. In our view, 
estimates should be updated only when significant changes 
have occurred. 
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IV. ORGANIZATION FOR THE PRODUCTION OF ESTIMATES (pp. 80-91) 
The current system for producing estimates gives each NIO 
a larger amount of authority and responsibility than any one 
individual had under the previous system, and it encourages and 
requires a close working relationship between the NIO and the 
main producers and users in his area. 

Both of these results were highly valued by most of the 
users who commented on the subject. Some of those interviewed, 
however, felt that the present system places too heavy a burden 
on the NIO, and that it suffers by lack of provision for col- 
legial review, as once supplied by the Board of National 
Estimates . 

One benefit of the present system, in the eyes of some 
producers, is that it puts the drafting responsibility on 
analysts who deal with a subject on a day-to-day basis. 

Others believed, however, that the analysts' lack of expe- 
rience in estimative writing and the conflicting demands on 
their time from other tasks often resulted in a lower quality 
estimative product. 

Changes in Current Practices (pp. 89-91) 

Efforts have been made to respond to some of the criti- 
cisms, including the authorization of a panel of outside 
consultants to review estimative drafts. Without offering a 
detailed blueprint, the study team believes that additional 
steps should be taken. These are: 
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--to establish a body of generalists to serve 
as an Internal collegial review group; 

--to establish a small group of analysts to 
do the initial drafting of estimates as a 
full time task; and, 

--to activate the authorized panel of outside 
consultants . 

These steps would help insure high quality and the 
thorough and complete treatment of a subject. They would 
provide an additional means of getting objective and dis- 
interested national estimates, and would highlight the 
primacy of the DCI. They should not be allowed to impinge 
on a strong virtue of the present system: its improved 
ability to maintain contact with the consumer and to 
insure the pertinence and responsiveness of the estimative 
product to the consumer's needs. 

V. THE USER'S ROLE IN ESTIMATES (pp . 92-102) 

If estimates are to be useful and relevant, producers 
must clearly understand the needs of users. Such under- 
standing can best be acquired by direct communication on 
matters of scope, timing and the issues to be addressed. 

An effective dialogue between producers and users would 
seem to require: 

--clear evidence of interest by the President and 
senior policy makers in the use of estimates; 

--a recognized procedure for fitting estimates into 
the national security decision process; and, 
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--active participation by senior policy and intelli- 
gence officials. 

To date, these conditions have been met only partially and 
sporadically. The National Security Council Intelligence 
Committee (NSCIC) was set up in 1971 to provide guidance by 
users on their needs and to evaluate intelligence products, 
but proved ineffective. Since it was abolished in 1976, there 
has been no formal mechanism for user-producer exchange. 

Despite what the NIOs have done to bridge the gap, intelligence 
production and collection are still determined more by what 
the producers think is needed than by direct requests or by 
guidance from users. 

Some believe that intelligence analysts should maintain 
a certain remoteness from decision makers to keep intelli- 
gence untainted by policy pressures. But most users and 
producers took a different view, considering a close rela- 
tionship mandatory--its primary benefit being a clearer, 
more realistic appreciation by each of the other's capabilities, 
limitations and needs. 

Providing for Closer Contacts (pp. 97-102) 

The major obstacles to closer user-producer relations 
are lack of time, physical separation, and a view among some 
policy makers that the producers of intelligence are well 
enough informed and sufficiently competent to determine on 
their own what users want and need. Most users agreed, 
however, that efforts to improve communication are desirable. 
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--One solution sometimes suggested is the creation 
of a forum such as the NSCIC, but without its 
liabilities. This course is worth trying, but 
not a great deal should be expected of it. 

It is quite clear that the improvement of communication 
will depend mostly on the efforts of individuals, and that 
the main burden will remain on the producer. 

--Several policy makers stressed the importance 
of developing personal relationships of mutual 
confidence with intelligence producers; 

--Users should do all they can to insure that 
the producer is aware of trends in policy 
and to understand what estimates can and 
cannot do; producers must help them acquire 
this understanding and absorb what is 
readily knowable about policy concerns; 

--More tours for selected intelligence officers 
in policy offices would be helpful. 


VI. THE FUTURE ROLE OF ESTIMATES (pp . 103-107) 

As its power and self-sufficiency become more circum- 
scribed, the U.S. will be increasingly dependent on accurate 
estimates of the possible plans and actions of its adversaries 
and friends. Thus, estimates will have a highly useful role 
for the foreseeable future and should get high priority in 
the overall intelligence effort. 

Estimates will be more difficult to prepare in the 
future. With the growing complexity and interrelatedness 
of the world's military, technological, economic, political 
and social affairs, it will be harder to understand and to 
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foresee crucial developments, to be clear and precise, and 
to phrase estimative judgments in simple declarative sentences. 
The likely shrinking of intelligence resources will be coupled 
with a requirement for estimates on a wider variety of sub- 
jects aimed at more and different consumers, including 
Congress and perhaps even the public. 
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I 

SETTING THE STAGE 

National Intelligence Estimates (NIEs) are almost as old 
as "central intelligence" itself, having been systematically 
produced by the United States Government since 1950. In 
October of that year, General W. Bedell Smith became Director 
of Central Intelligence (DCI) and established an Office of 
National Estimates (ONE), together with a nine-man Board of 

National Estimates under / \ 25X1 

University . 

In the early years, estimates tended to be short, spare, 
narrowly focused, and concerned with events in a very short 
time frame. NIE #1, written in November, 1950, was entitled 
Pros pect for Communist Armed Action in the Philippines During 
November. Others written in the first year or two were very 
cautious about looking more than a month or two ahead. Esti- 
mative judgments were often tendered with little of the under 
lying rationale provided in the text. 


Probably the first "national estimate" was prepared under 
the direction of Admiral Roscoe H. Hillenkoetter as DCI in 
March 1948. Its subject was Soviet intentions in the Berlin 
crisis, and its production was a milestone in interdepartmental 
cooperation and a forerunner of th e system adopted in 1950. 

' 1 'in Studies 


See "March Crisis 1948, Act I,' 
in Intelligence, Vol. X, No. 4 


by 

(Fall 1966), p. 13, 


25X1 
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As analytical capabilities and the data base grew, 
estimates also changed. Greater emphasis was placed on the 
presentation of data and on the development of the argument. 
The forces at work in a situation were discussed and trends 
identified and evaluated. Estimates were produced whose 
objective was to educate and to provide background. 

These changes were stimulated in considerable part by 
changes on the demand side. From the beginning, the market 
for estimates went well beyond the top policy makers and 
their senior aides, and it grew greatly with the passage of 
time. It came to include readers with widely varying back- 
grounds and requirements. Many of them required sophisticated 
analysis, and became increasingly unwilling to accept the 
judgments of others without a backdrop of fact, argumenta- 
tion and analysis. 

In this respect, a major turning point occurred with the 
advent of the Nixon administration in 1969. Skeptical from 
the first about the estimative product, its senior officials 
demanded a complete justification of the judgments rendered, 
particularly for military estimates. They also required that 
sufficient data and analysis be presented so that the reader 
could reach his own conclusions from the material presented. 

Estimates had their clearest formal role in the decision- 
making process during the Eisenhower administration, when the 
National Security Council (NSC) was a more structured and 

20 

CONFIDENTIAL 

Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 



Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 


CONFIDENTIAL 


important policy making body than at any time before or since. 
The NSC Planning Board was the place where NIEs were brought 
together with policy inputs and combined for use by the NSC. 

Not every NIE had an impact on policy, but there were clearly 
defined institutional linkages connecting intelligence with 
the policy process. The potential of those arrangements was 
not fully realized, however. Estimates were not consistently 
read by high-level officials, and there is not much evidence 
that they had a major influence on policy. 

With the decline in the role of the NSC machinery under 
the Kennedy and Johnson administrations, the systematic 
structured connection of estimates to policy was reduced. 
Nonetheless, the true heyday of estimates may have been in 
the early 1960s. It was then that the White House and impor- 
tant Cabinet members asked for and used more estimates in 
formulating policy than at any other time. The DCI and senior 
intelligence officers had access to the President and other 
top policy officials, and the DCI personally played a crucial 
role in inserting estimative products into the policy process. 
Also, at about that time rapidly expanding technical collection 
capabilities began to pay rich dividends, which were reflected 
in more comprehensive and sophisticated estimates, especially 
on technical and military subjects. 

The influence and importance of estimates began a decline 
in the mid-1960s. Complaints about shortcomings in estimates 
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have been voiced with increasing frequency, particularly 
since 1969. Although the themes had been heard in one form 
or another from the very beginning, the complaints built to 
a crescendo in the late 1960s and early 1970s. There appear 
to be a number of reasons, including internal governmental 
strains and divisions over Vietnam, a lack of access and 
influence with the White House by the DCI, and a belief by 
consumers that estimates were increasingly out of touch with 
the needs of the executive. 

Partly in response to these complaints, DCI William Colby 
abolished ONE and the Board of National Estimates in November 
1973, and established the system currently in effect. Overall 
responsibility for producing policy-relevant estimates, and 
for maintaining close contacts with producers and consumers 
of intelligence, was vested in some dozen senior National 
Intelligence Officers (NIOs) , who reported to the DCI and 
were responsible for particular geographic regions or sub- 
stantive specialties. The actual drafting of estimates was 
diffused: NIOs could call on intelligence production com- 
ponents anywhere in the community to furnish drafters, but 
in practice most drafting of non-specialized estimates fell 
to officers in CIA's Directorate of Intelligence. 
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II 

ESTIMATES AS AN INSTRUMENT OF POLICY SUPPORT 

In setting about to study the role of estimates and in 
evaluating their success in supporting policy, it is first 
necessary to understand the extant doctrine of estimates- - 
that amalgam of directives, pronouncements, and history 
which describes what estimates are supposed to be and do, 
and for whom they are supposed to be written. This chapter 
begins with that task. It follows with a discussion of what 
consumers say about their use of estimates and the value they 
place on them in order to see what differences exist between 
doctrine and fact. 

The Traditional Doctrine 

Part of the traditional perception of the role of 
estimates resides in the word itself. As distinct from 
reports and studies, estimates were to be forward-looking 
rather than historical and descriptive. Their ultimate 
purpose was to make judgments about the shape of the future 
on matters of concern to U.S. policy--that is, to estimate. 

National estimates were intended to have a position 
of primacy among the various intelligence products prepared 
for decision makers. Estimates were to be the best and most 
authoritative appraisals available on subjects of national 
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security concern. As such, they were to be national products 
in three senses: their subject matter was to be suitable for 
assisting in the formulation of national security policy; 
their audience was to be at the top levels of government- - 
including the President, members of the NSC, and similar 
officials; and the process of producing them was to engage 
the entire intelligence community, so that its combined knowl- 
edge could be brought to bear. Nonetheless, the estimate was 
to remain the product of the DCI, who was charged by statute 
with the production of national intelligence. Other partici- 
pants in the process could disagree with the DCI, could 
attempt to persuade him to change his mind, and could express 
their dissents in the estimate itself, but they could not 
impose their views on him, or put him in the position of 
being a dissenter from the majority view. 

The process was intended to achieve two other results. 

The first was quality sufficiently high to engage the most 
senior policy makers and to illumine their concerns. The 
second was objectivity and disinterestedness. This was to 
be achieved, in the first instance, by involving the entire 
community, rather than any single agency, in the production 
of an estimate. Beyond this, as the DCI's estimate, it was 
separated from the purview of intelligence agencies whose 
parent organizations had policy responsibilities, and assigned 
to an individual who did not, and whose responsibilities were 
to the NSC and to the President. 
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Ho w Estimates Fared 

Against these standards, estimates did not fare well in 
the view of the users interviewed. Estimates were often found 
wanting in quality and in relevance to the concerns of the 
policy maker, and they did not play the role that had been 
envisaged for them in the formulation of policy. Although 
some individuals were high in their praise, and others found 
estimates useful in one or another respect or in particular 
situations, most users had critical remarks to make. In the 
aggregate, the quality and utility of estimates were viewed 
as falling well short of the traditional ideal. 

The Negat ive Side . The first thing to be said is that 
most estimates do not reach their intended audience. If they 
are called to the attention of the President and NSC members, 
it is usually through memoranda written by subordinates who 
paraphrase the contents and add comments of their own. Some- 
times marked or annotated copies are sent along. Below this 
level, estimates are widely received and frequently read. 

Even here, however, their content is often absorbed indirectly, 
through briefings, conversations, and memoranda. 

As to the quality and utility of estimates, perhaps the 
most fundamental complaint was that they were not tied into 
the ongoing concerns of the policy maker. They were frequently 
criticized as irrelevant, and for showing unawareness of the 
specific policy problems facing the reader. As one respondent 
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put it, the producers of estimates doubtless wanted to make 
them relevant, but apparently did not know how to go about 
it. Another complained that on the big foreign policy issues, 
estimates very often overlooked some of his problems and did 
not hit on his operational needs. 

Such critics viewed estimates as documents which supplied 
general background and little more. For the most part, they 
placed little value on such background and could often not 
spare the time to read estimates. As one user put it, a lot 
of estimates wound up in his Saturday reading file, and were 
never read. Another allowed that estimates did perform an 
educational function for staff people, but he also said that 
they were never a guide to action. 

Estimates were frequently criticized for being insuf- 
ficiently analytical and overly descriptive. Many thought 
them lacking in boldness, and as more concerned with the 
comfortable present than with the unknown future. There 
were complaints about the "fuzziness" of their judgments 
about the future. One respondent said that many of the 
papers he got were "conservative"; the analysis was not 
taken very far, implications were not drawn, and there was 
a fear of making predictions. 

Skepticism was also expressed about the quality and 
usefulness of the predictions that were made. Many readers 
were simply unwilling to accept a "faceless" bureaucratic 
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product whose authors and their capabilities were unknown to 
them. They stressed the importance of the personal element 
in receiving and accepting advice, and of knowing something 
about the cast of mind and the thought processes of those 
who proffered it. They felt that a written product would 
inevitably fare poorly in competition with aides and con- 
fidantes, unless it was skillfully laid out and of exceptional 
quality. 

Estimates were also faulted for insufficiently explaining 
their judgments and conclusions, and for not discussing what 
alternatives were considered and why they were rejected. 
Consequently, readers often felt that they were being asked 
to accept conclusions from "on high." Unpersuaded by analysis 
and argumentation, and lacking personal knowledge of the 
authors, many were unwilling to do so. 

Skepticism was fed by a widespread belief that estimative 
products contributed little that was unique in making judg- 
ments about the future. Most policy makers- -even the very 
senior--read considerable quantities of raw traffic, and 
consider themselves well informed. Many felt that they were 
in as good a position as the intelligence specialist to make 
estimates, and frequently in a better one, because of their 
greater first-hand knowledge of particular countries and 
leaders. This belief was far stronger on political subjects 
than on subjects of a scientific, economic or military nature. 
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Past errors also played a role. Some cited specific 
cases in which they felt that more accurate judgments could 
have been made, even given the hazards of estimating and the 
benefit of hindsight. Such remarks require careful inter- 
pretation, however. They concerned mainly the general impact 
of estimates (e.g., they were too bland or too reassuring), 
or inadequacies in the portrayal of broad forces, trends, and 
implications. There was a tolerance of failures to make 
accurate predictions of specific events, such as a change 
of government or the outcome of an election. Our respondents 
recognized that such predictions are difficult, and were 
disposed to forgive failures, providing the basic analysis 
had been sound, and pertinent factors had been considered 
and illuminated. Indeed, a sizeable number of our respondents 
felt that specific predictions of specific events should not 
even be attempted. 

The Positive Side . Some respondents took a strongly 
positive view of estimates and the estimative process, and 
advanced few or none of the criticisms set forth above. In 
effect, they felt that estimates met most if not all of the 
desiderata of the traditional doctrine. On the whole, they 
were less concerned than others about direct relevance to 
immediate policy issues, less operationally oriented, more 
interested in the "big picture," and more receptive to 
material which educated or provided background. They placed 
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greater value, as one put it, on getting a "tapestry," and 
on the usefulness of an informed and disinterested view from 
outside the policy process. 

Two of the three former cabinet members we interviewed 
fell into this category. One described the role of estimates 
in supporting policy as fundamental, and valued them for their 
objectivity and professionalism. The other said that estimates 
helped the policy maker to go through a mental checklist of 
what he should consider before making an important decision. 

He believed that they gave the policy officer a lot of infor- 
mation that he could tuck away and summon when he needed it 
most, and when he had the least time to acquire it. 

The rest of those with a positive view fell into a dif- 
ferent category. Not as sweeping or categorical in their 
praise as the group just discussed, neither were they 
troubled by the kinds of deficiencies perceived by the more 
critical. In a sense, they had more modest expectations for 
estimates, and these expectations were on the whole realized. 
For example, one senior military officer told us that esti- 
mates do pretty well in covering the subjects he is inter- 
ested in and said that he is not too unhappy with the way 
things are, perhaps because he expects less than others. 

Another respondent said that he found estimates useful on 
specific occasions and in certain contexts, and the work 
objective and thorough. 
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Many respondents found estimates useful because they 
pulled together in one place all that was known about a sub- 
ject. This was meant in one or both of two senses. First, 
the estimate was perceived as a product which took into 
account all the available sources on a particular subject. 
Second, it was valued as the work of a body of experts from 
outside the policy process, who had given considerable thought 
to the subject. These qualities were found useful by many who 
faulted estimates on one or more of the grounds mentioned above. 

In the same vein, estimates served for some as a kind of 
security blanket. That is, by providing a systematic and com- 
prehensive treatment of facts, trends, and issues, they helped 
assure the reader that he had considered all the factors 
bearing on a problem. If an estimate confirmed his own 
analysis and conclusions, so much the better. It could 
then be used, if need be, to convince a skeptical superior 
or a doubting colleague. Indeed, some estimates were requested 
for the very purpose of persuading colleagues and superiors of 
the rightness of the requestor's views, and were used for that 
purpose . 

Other Finding s. During our interviews, we solicited 
views on the objectivity of estimates, and the presence or 
absence of bias. This is a matter of considerable importance, 
particularly in view of the virtues traditionally ascribed to 
the product and the process. 
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On this score, estimates fared well. Indeed, a few 
respondents considered the objectivity of national estimates, 
and the fact that they were prepared outside the policy process, 
as among their principal virtues. Included, as noted, was a 
former cabinet member. Numerous others praised the balance 
and professionalism of the product. Some who did so had been 
critical of estimates on other grounds. For example, one 
individual observed that most of what he read was balanced 
and professional, even though these qualities led to what he 
called blandness and to a lack of focus on the issues which 
count . 

Some of our respondents did say that they had observed 
bias in estimates, either currently or in the past. With two 
exceptions, however, they viewed this as a comparatively minor 
problem, rather easily discerned and adjusted to, and as the 
consequence of human imperfection rather than of deliberate 
intent. As one person put it, the record varies, everyone is 
a prisoner of his own stereotypes, but the intelligence com- 
munity makes a reasonable effort. Another said that it was 
not possible to get totally away from subjectivity or from 
vested interests; he said that he did not currently see hidden 
policy biases, although he felt that vested interests had shown 
through in the past. 

In only two cases did an alleged lack of objectivity bulk 
large in the comments of our respondents. In one case, a former 
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senior official felt that estimates frequently and with intent 
superimposed viewpoints not derived from the facts, or allowed 
facts to be overruled by the conventional wisdom. In the other, s 

our respondent in effect imputed bias to estimates because their B 

conclusions did not agree with his, and, as far as we could tell, i 


for that reason alone. 

We attempted to get a sense of whether users of estimates 
perceived any differences in the quality and utility of esti- 
mates as the result of the change in the method of producing 
them made in 1973. This proved difficult, partly because of 
turnover at the policy making level and the unfamiliarity of 
many with the process and the product before 1973. There was, 
however, a fair degree of consensus on two points. The first 
was that the NIO system had produced modest improvement in 
the relevance of estimates and somewhat greater improvement 
in the flexibility and responsiveness of the system. Con- 
versely, estimates were thought to be more uneven in quality 
than before 1973, a result which some attributed to the new 
procedures for drafting them. 

The interviews revealed what appeared to be systematic 
differences in reactions to different kinds of estimates. 

The so-called "hard" estimates -- those on military, scientific 
and technical, and economic sub jects- -tended to get a better 
reception than those on politics or foreign policy. This was 
apparently not a function of differences in intrinsic quality. 
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but rather of differences in the policy maker's familiarity 
with the complex and difficult subject matter, his ability to 
do his own analysis, and to reach his own conclusions. Also, 
many judgments in military estimates are computational or 
represent the extrapolation of comparatively hard information. 

By comparison, judgments on political matters, or on intentions, 
seem fuzzier, less precise, and less supportable, and often are. 

Another finding is advanced with caution. On the whole, 
there appeared to be more satisfaction with estimates at the 
country director or equivalent level than at higher levels. 

Caution is necessary because our sample at the country director 
level was small. We mention this matter only because the impli- 
cations are considerable, and because it may warrant attention 
at some future time. Perhaps higher level policy makers need 
a product so sensitive to their particular concerns that it is 
virtually impossible for a producer many steps removed to 
satisfy them effectively, without new arrangements for exchanges 
on needs and capabilities. The country expert and the producer, 
by contrast, may more nearly share perspectives, as well as 
views on the evidence and the conclusions that can be drawn 
from it. 

We failed to find among consumers certain concerns that 
criticisms of estimates in recent years had led us to expect. 

For example, consumers did not consider that they were getting 
confused signals as the result of the proliferation of intelligence 
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publications that cross their desks, estimates among them. 

The existence of departmental as well as national estimates 
on some matters also did not seem to present a problem. In 
general, users seemed to appreciate getting a variety of view- 
points, so long as they were analytical and interpretive and 
not just a duplicate catalogue of facts or narrative description. 

Similarly, there did not appear to be much concern with 
the present style of estimative writing, or with the lack of 
some explicit scale of probabilities. The use of traditional 
expressions such as "probably," "likely" and so forth seemed 
satisfactory, though they admittedly do not convey the same 
meaning to all readers. The experimental efforts to provide 
explicit quantitative rating scales for probability apparently 
have not struck any very responsive chord with users. They 
were never mentioned as examples to be emulated. 

Why the G a p? 

Why was there such a disparity between what estimates are 
supposed to be and do and what was said about them by those 
interviewed? The answer must begin with a mea culpa . Much 
of the problem lies with deficiencies in the product, and in 
the failure to provide the users of estimates with the kinds 
of product they believe they need. 

Much of what was said above speaks for itself. It seems 
clear that estimates have frequently not understood the policy 
question, that they have been insufficiently venturesome, and 
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that they have not done enough about discussing the implications 
of their analysis and conclusions for the U.S. and for U.S. 
policy. Estimating is a difficult and hazardous art, and the 
temptation to be descriptive and current, rather than analytical 
and forward looking, has not always been resisted. 

The answer does not end with a mea culpa , however. There 
are a number of other circumstances which account for the 
failure of estimates to achieve the eminent position contem- 
plated for them. Individually and collectively, these are of 
considerable importance; what is more, they explain in part why 
the quality of estimates has frequently left something to be 
desired . 

Some Unrealized Assumptions . A basic difficulty is that 
the doctrine for estimates represents an ideal, which puts an 
unrealistic burden on the DCI and the intelligence community. 
Specifically, the doctrine rests on some unstated and unrealized 
assumptions, which have borne little resemblance to reality, 
and which should not have been expected to. These are: 

1) that intelligence in general, and estimates in particular, 
would have a major influence on the formulation of national 
security policy, if not the major influence; 2) that policy 
makers would seek and welcome the contributions of estimative 
intelligence even when they cast doubts on current policy; and 
3) that, consequently, the relationship between national policy 
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and national intelligence would be close and intimate, and 
communication on capabilities and needs free and complete. 

As to the first of these assumptions, national estimates 
have played only a modest role in the formulation of policy. 

Almost certainly, this was an inevitable development. National i 

security policy is not directly driven by facts, analyses, and 
judgments, and knowledge does not lead to action in any simple 
way. What also come into play are competing bureaucratic 
pressures, a wide range of domestic considerations, the con- 
cepts, perceptions and stereotypes held by the policy makers 
themselves, and a host of essentially political inputs. 

Policy making is a complex and often lengthy and untidy 
process. Many policy decisions - -perhaps most of them--are I 

not made at a particular time or place, as the result of a 
particular act, or as the product of a formalized, established 
process. Policy may emerge incrementally, or as the result of I 

a series of specific decisions, no one of which appeared por- f 

tentous at the time it was made. It may be generated by efforts I 

to prepare a speech or a statement to the press. Under these I 

circumstances, there are some real and difficult questions I 

about how estimates are to impact on the policy process, and 
at what point. 

Intellectually, the producers of national estimates have I 

long recognized the limits of their role, as these limits have ? 

been borne upon them by experience. Yet, the older notion I 
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has been hard to shake, so firmly has it become part of the 
mythology . 

As to the second assumption, it was overly optimistic to 
assume that policy makers would represent a ready and even 
eager market for estimates. As noted earlier, policy makers 
read raw intelligence traffic, consider themselves well informed, 
and often believe that they can do as well at estimating as the 
intelligence specialist, if not better. Moreover, because of 
the past accomplishments of intelligence itself, policy makers 
have access to a corpus of fact, analysis and judgment on 
national security issues from a national vantage point--a 
resource taken for granted now, but not available in earlier 
years. Further, they have access to a continuing stream of 
other analytic and estimative material, including departmental 
estimative products. Many users attach no special value to a 
coordinated, national product as compared with a departmental 
issuance, and some prefer the latter. 

Additionally, if national estimates do their job, they 
may wear out their welcome. Thus, one of the functions of 
estimates is to help policy makers think about their problems 
and to address, and if necessary to challenge, their assumptions 
and judgments. In the process, an estimate may say things the 
policy maker does not wish to hear and which make life more 
difficult for him. On divisive issues, or where the political 
or budgetary stakes are high, producers of estimates can be 
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cast in the role of interlopers and troublemakers, and they 
run the risk of being considered not on the team. Much depends, 
of course, on the attitudes existing on both sides, and on the 
presence or absence of mutual confidence and trust. 

Finally, the idealized relationship involving free and 
frank interchange between producer and user does not exist, 
and events of the past few years made it less of a reality 
than it used to be. There is, moreover, a conspicuous absence 
of structure for systematically insuring that estimates are a 
part of the policy process, and that they can make whatever 
contributions their own virtues allow. The reference here is 
only in part to a formal bureaucratic mechanism; more funda- 
mentally, it is to a concept of how estimative intelligence 
and policy are to complement one another, and how information 
on capabilities and needs is to be exchanged. 

Other Reasons . The failure of estimates to play their 
prescribed role is in part the result of two other circum- 
stances. First, the foreign policy establishment is, on the 
whole, highly operational. This means that it is concerned 
with the policy matters immediately before it and that these 
are often short term and highly specific in nature. Frequently, 
users judge an estimate by whether it meets such operational 
needs; if it does not, it is deemed irrelevant, and it remains 
unread, or is read and ignored. For example, one user complained 
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that a paper on a country of interest to him had not established 
the relationship of what was said to certain issues then under 
negotiation with that country. 

One solution, of course, is for the intelligence estimator 
to do the fast footwork required to inform himself of what is 
needed and to supply it. Under the NIO system (described in 
Chapter IV), attempts have been made, with some success, to 
get a better feel for the requirements of users and to devise 
more flexible and responsive estimative formats. The Inter- 
agency Intelligence Memorandum (IIM) is one such form. Typically, 
the IIM addresses matters of current concern to the contemplated 
users and it can be more quickly and easily coordinated than an 
NIE, because it does not require a formal sign-off by the 
various intelligence chiefs. 

Although Operational relevance is important, there is another 
aspect to the matter. Estimates have been visualized by those 
who produce them not merely as products which supported day-to- 
day operations, but as contributions to the formulation of 
basic, long term national security policy. They were to assist 
in developing concepts and in establishing broad, general direc- 
tions for policy. Accordingly, it was at one time thought that 
a major consumer of national estimates would be the Policy 
Planning Staff of the Department of State, performing in the 
role that was originally contemplated for it, and not in the 
operational role it has often subsequently assumed. From 
this perspective, it is not enough to say that estimates are 
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frequently deemed irrelevant to policy requirements. Perhaps 

the difficulty is with the way policy has been formulated, and 

with the atrophy of the functions and institutions that esti- I 

mates were supposed to serve. 

1 he other circumstance is that the reception given esti- I 

mates is very much affected by the attitudes which develop I 

among those who receive and use them. These attitudes tend to 6 

be set at the top. Thus, if intelligence is welcomed and read 1 

at the top, and taken seriously there, it will be read and I 

taken seriously throughout the government. The reverse is e 

also true. It became clear, beginning in 1969, and perhaps s 

earlier, that intelligence in general and estimates in par- I 

ticular were not held in high esteem at the top, and in a I 

degree quite out of proportion to what seemed justifiable. I 

There was deep suspicion and distrust, not only of the esti- § 

mates, but of those who produced them. This attitude was f 

bound to permeate much of the executive branch, and it did, f 

and the production and use of estimates has yet to recover fr 

from its effect. 

The Ma rket for Estim ates 

The comments of our respondents on the quality and value I 

of estimates laid the basis for another category of questions. f 

These concerned the nature of the market for estimates. Was 
there a continuing requirement for national estimates? If fc 


40 

CONFIDENTIAL 

Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 



Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 


CONFIDENTIAL 


so, for what kind? Was their main value in their analysis, in 
their judgments about the unknown future, or in something else? 
Where was the market? Should estimates be written for policy 
makers at the very top, or for some other audience? 

On the broad question, the users' view was that national 
estimates do have a role to play in the policy-making process. 
This was true even of those who had been most critical of the 
past product; their problem was not with the legitimacy or 
usefulness of the estimative function, but with the way they 
considered it to have been carried out. Our respondents 
believed, to be sure, that policy makers would rely heavily 
on their own judgment, and on the judgment of trusted aides. 

But they also saw a place for estimative products which do 
certain categories of things. 

Before getting into what these things are, a brief digres- 
sion is in order. As some respondents suggested, the require- 
ment for estimates is probably greatest when a new Administration 
takes office and for a period of time thereafter. A new group 
of top policy makers needs to inform itself, and to familiarize 
itself with the issues. Old concepts and policies must be re- 
examined and new ones formulated and tested. Once the new team 
has absorbed the larger picture, and established the main lines 
of policy, the requirement lessens. At the extreme, as one 
respondent put it, estimates are not missed when things are 
set and there are no changes in direction. 
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What Should Estimates Do ? As to what estimates should t 

strive to do, the responses depended, to a certain extent, t 

on the kind of estimate involved. The role of estimates in I 

sorting out the facts, for example, was more highly valued f 

on military or technical subjects than on matters of internal f 

politics or foreign policy. This reflected in part the greater I 

detail and complexity of the issues, and in part the lesser I 

confidence of users in their own ability to do the job. I 

The responses covered a fairly wide range, but they again | 

tended to cluster around recurring themes. One was that esti- I 

mates should help to identify policy problems, and to pose I 

Issues and questions about them. "Identify" is the key word. f 

That is, estimates were deemed useful if they surfaced prob- I 

lems which had not occurred to the reader, or were not cur- I 

rently in the forefront of his attention, and if they helped I 

him to understand what their policy impact might be. An f 

example would be a paper which foresaw basic changes in the I 

Sino-Soviet relationship, with a consequent impact on U.S. I 

relations with both countries. f 

Estimates were also valued when they could sort out the l 

facts and issues in a highly complicated situation. Here the ! 

key word is "complicated." What our respondents meant was f 

that in some cases, the facts are so elusive, so apparently I 

contradictory, or so fast changing, as to be unintelligible 
to anyone whose principal exposure to them is from reading r 
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raw traffic or perusing current intelligence publications. In 
such circumstances, they saw a role for estimative intelligence 
in sorting out the facts, the issues they raised, and the pos- 
sible consequences for policy. An example would be a paper 
written in the early stages of the civil war in Angola, which 
described the strength and composition of the opposing forces 
and the state of play in the political arena in that little - 
known country, before tackling the question of probable future 
developments and their impact on the U.S. 

Another perceived function was that of identifying and 
evaluating the forces at work in a situation. In essence, 
this is an aspect of analysis. The interest was in what was 
driving events- -in the causal forces whose continuation or 
manipulation would affect the final outcome, and in the inter- 
play of those forces. An example would be an examination of 
the internal political and economic forces which accounted 
for the position taken by various foreign countries in inter- 
national monetary negotiations, and an appraisal of how these 
forces could be manipulated or circumvented to serve U.S. ends. 

Closely related, there was also a desire for analysis 
of trends and their probable outcome. That is, there was a 
requirement for estimating. The focus was not, however, 
on specific estimates of specific future happenings. It was 
rather on how events could be expected to develop as the 
forces at work played themselves out, and how changes in the 
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forces at work would affect the ultimate outcome. An example 
would be an examination of probable future developments on 
the Italian political scene and their impact on Italian policy 
toward NATO, the Common Market and other institutions of 
interest to the U.S. 

Still another perceived function was that of judging the 
consequences of ongoing developments for U.S. policy. This 
meant, in effect, supplying the "so what" of that which had 
come before. It was perceived as the step which converted 
analysis into a coherent statement of what the interplay of 
forces and their probable development meant for U.S. interests 
and U.S. policy. It included the identification of forces and 
trends which could undermine the premises on which U.S. policy 
was based. There is no need to cite examples; it would be a 
rare estimative product which did not contain this dimension, 
as the above examples suggest. 

Finally, estimates were valued when they were able to 
judge the reactions to U.S. policies or actions, ongoing or 
contemplated. Here, U.S. policy was taken as given, and the 
estimative input was not to its formulation, but to a judg- 
ment of its effects. In part, our respondents had in mind 
so-called contingency estimates, which attempt to describe 
what would happen if the U.S. were to do this or that. An 
example would be a paper which examined the reactions abroad 
to a contemplated arms control proposal, and which assessed 
its prospects. 
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Apart from these more or less recurring themes, certain 
users saw other values in estimates. Worth mentioning here is 
the interest expressed by some in getting long term analyses 
which they did not have time to do for themselves. Almost 
invariably, they had in mind a division of labor, rather than 
the performance of a function which they thought estimative 
intelligence particularly well equipped to carry out. Such a 
requirement has interesting implications, for it views estimates 
coming from outside the policy process as a mere convenience to 
a policy maker who is too busy, and not as publications with 
a special contribution to make. 

There were several reasons for the relative lack of 
interest in specific predictions. Some thought that estimates 
placed too much emphasis on prediction at the expense of 
analysis and the development of conceptual frameworks. 

Others simply placed little value on the prediction of events 
such as coups, elections or changes of government. Still others 
valued predictions not for their own sake, but because they 
forced estimates to move away from the factual and the descrip- 
tive and to focus on analysis and the general shape of the 
future. Overall, there appeared to be considerable skepticism 
about the capacity of the intelligence community to foresee 
specific future events, going both to the inherent difficulties 
and to doubts that it possessed any special qualifications. 

Taken as a whole, what is most striking about the com- 
ments received is their reaffirmation of the traditional 


45 

CONFIDENTIAL 


Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 



Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 


CONFIDENTIAL 

function of the national estimate. Emphasized were the same 
desiderata set forth in the doctrine: a focus on the analysis 
of forces, trends, and implications for the U.S., in a con- 
text that is analytical and forward looking rather than 
descriptive and current. 

For Whom Should Estimates Be Written ? The nature of the 
audience for estimates is a matter of considerable importance. 

It affects the way they are organized, their level of detail, 
and how they go about presenting facts, analyses and conclusions. 

Some of those we interviewed thought that estimates 
should be prepared for readers at the very top--the President, 
members of the NSC, and others at the cabinet level. This was 
usually not out of confidence that estimates would regularly 
be read there, but in the belief that estimates aimed at such 
customers would catch others along the way. There is some- 
thing to this view, but it misses an important point--that 
the perspectives and needs of those at the top are not neces- 
sarily the perspectives and needs of those who support them. 

It is almost certainly too much to expect that busy 
policy makers at the highest levels will read estimates 
regularly. Nonetheless, they will be customers under some 
circumstances. Estimates on particular subjects will from 
time to time commend themselves, such as those on Soviet 
strategic objectives and military forces or those making a 
significant contribution to the conduct of important 
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negotiations. So will products of exceptional quality and 
insight. Particular individuals will value estimates for a 
variety of reasons- -because they come from outside the policy- 
making establishment, because they inform and illumine on 
matters of concern, or because they contain agreeable con- 
clusions useful in the policy debate. 

The estimates most likely to be read at the highest 
levels, however, are those which the DCI believes should be 
read there, and which he urges upon senior policy makers. 

It is one thing for an estimate to be issued in the name of 
the DCI. It is quite another for the DCI, as a trusted senior 
aide of the President, to commend it, and to press it upon 
others. If the DCI has the necessary access and enjoys the 
confidence of the President and his chief aides, then any 
given estimate has a far greater chance of being read at the 
top, and consequently, throughout the government. "Any given 
estimate" is the way to put it, because the use of the DCI's 
position, prestige and access, if it is to be useful and 
effective, cannot become routine. Nor can it be fully 
effective unless the DCI involves himself in the process 
of producing estimates, so that he can sell what is truly 
his, and help insure that estimates are relevant and of high 
quality . 

The reception afforded estimates at the top will also be 
affected by the attitudes toward them that exist or develop 
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there. We have already suggested that over the past few years, 
there was a disposition to discount the importance of the 
estimative function, and a deep skepticism about the value 
of the product. In this respect, too, the DCI plays a crucial 
role. His personal qualities, his access, and the confidence 
that the President has in him can help establish a receptivity 
for the product, which its own qualities would then have to 
sustain . 

Most of our respondents felt that estimates should be 
aimed at the Assistant Secretary of State, White House Staff, 
or comparable level, and we agree. These are the officials 
who set the terms for the debate on matters of national 
security policy by formulating the options and alternatives. 
They are the trusted aides, referred to earlier, who enjoy 
the confidence of policy makers at the highest levels, and 
who are turned to for advice. On many problems, they are 
the highest level which combines expertise with the power to 
act. Readers such as these are able and knowledgeable. With 
access to the same material as is available to the intelli- 
gence community, they feel able to do their own analysis, 
and to reach their own conclusions about the shape of the 
future. They will not accept unsupported judgments, in 
the form of "we believes" or otherwise. 

How Should Estim ates Be Written ? If estimates are to 
appeal to this audience and be useful to it, they must be 


48 


Approved For Release 2002Wl/Wf<®^bp80-00630A000300040001-3 



Approved For Release 2002/11/04 : CIA-RDP80-00630A000300040001-3 


CONFIDENTIAL 


prepared with these considerations in mind. As we gleaned it 

from our interviews, they must: 

--emphasize analysis rather than description, 
and carry the analysis to the edge of the 
unknown ; 

--show the relationship among data, analysis 
and conclusions and the thought processes 
involved, so that the reader can make his 
own journey to a different destination if 
he believes it necessary; this also en- 
hances the credibility of the product and 
helps to compensate for its "facelessness ; 

--explain clearly what issues were most con- 
tentious at the coordinating table, and why; 

--discuss what was discarded and why; this 
not only helps build confidence in what 
was retained, but may persuade the reader 
that his views, if different, might better 
be discarded; 

--state any differences of opinion, and on 
what they are grounded, and avoid the 
temptation to consider consensus desirable 
per se ; 

--state the implications of the analysis and 
conclusions for the U.S. and for U.S. policy, 
and 

--describe the continuity or change in the 
current estimate as compared with its 
predecessors, particularly the change, 
and set forth what earlier material is 
now considered wrong, and why. 

One difficulty with this list is that it makes for longer 
papers, and longer papers tend not to be read. We mentioned 
this dilemma to a number of those we interviewed, but beyond 
agreeing that it existed, they had little to offer. It is 
also true, however, that papers which slight the above 
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desiderata will probably not be read either. Thus, the pro- 
ducers of estimates will have to find ways of being complete 
and persuasive, yet readable and spare. To the extent that 
length cannot be avoided, quality will compensate for it, 
among the readers we are now talking about. 

At higher levels, the problem is more serious. To pro- 
vide for the occasions when estimates will be read there, they 
must contain appropriate syntheses and summaries, with invita- 
tions and directions for dipping into detail if such is desired. 
Summaries are also useful to readers at all levels who for one 
reason or another do not desire detail in particular cases. 

The importance of an effective summary cannot be overstated; 
it was a point made to us, without any prompting, by some of 
the most senior policy makers with whom we talked. Of this, 
more is said in the following chapter. 
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III 

IMPORTANT ELEMENTS IN THE PRODUCTION PROCESS 
FOR NATIONAL ESTIMATES 


The preceding chapter discussed the views held by con- 
sumers on the quality and utility of estimates. Some consumers, 
as well as producers , also had views about specific aspects of 
the estimative process. In this chapter, those views are dis- 
cussed in detail, along with perspectives developed by the 
study team. 

Self- In i ti a ted Estimates 

Roughly half of all the national estimates presently pro- 
duced are initiated by the intelligence community itself, and 
most producers and users with whom we talked seemed satisfied 
with this fact. A very few felt strongly that work on an 
estimate should be initiated only at the specific request of 
a consumer, primarily to avoid the production of irrelevant 
or unneeded papers. The vast majority, however, believed 
that intelligence producers have the duty of initiating an 
estimate when they perceive the development of a significant 
problem for U.S. policy. We agree, and would add that this 
appears particularly important at times when the policy making 
apparatus is preoccupied with some special problem, such as a 
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major and prolonged international crisis, or at the beginning 
of a new Administration. 

Two important corollaries should be kept in mind. The 
first is that the most useful estimates seem to have been 
produced in response to specific requests by policy makers 
who really wanted and needed them. Cranking out a lot of 
self -generated estimates does risk overloading the circuit, 
thereby diluting user receptivity, and wastes time and money. 
Also, a self-initiated product typically has a harder time 
being accepted by busy consumers psychologically unreceptive 
to thinking about issues ahead of their time, particularly 
if the product arrives unanticipated. The second corollary, 
therefore, is that any estimate generated by the intelligence 
community stands a better chance of being read and used if 
it is discussed fully from the outset with its intended 
recipients. Prior to doing this, the producers should focus 
carefully on their proposed effort to determine whether it 
will be perceived by users as related to ongoing or potential 
U.S. policy concerns. If the proposed product appears to 
have little relationship to such concerns, it is not likely 
to be worth the effort. 

T er ms of Refe r e nee 

On any estimate, there ought to be clear agreement between 
producer and user on the major questions to be addressed. The 
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device for securing this agreement is the "terms of reference" 
for the estimate. 

The production of an estimate generally begins with the 
elaboration by the NIO or an interagency group of the topics 
to be explored and the scope of the estimate. Currently, 
producers take part in the process, and the policy customer 
may also, either directly or through his agency's intelli- 
gence organization. To the degree that producer and user can 
initially agree on the questions to be answered, and can 
throughout the production process jointly further refine the 
scope of the paper, the resulting estimate can be made more 
relevant to the specific needs and interests of the main 
recipients . 

In general, the more direct the communication between 
producers and users on the scope of a paper, the better. A 
formalized, regular procedure providing for ample initial and 
subsequent consultation between estimators and policy making 
customers should be instituted. This process should not, how- 
ever, be allowed to become so formal and bureaucratic as to 
greatly lengthen the time that it takes to do an estimate, 
for such delays would themselves detract from the utility of 
the product. 

Encouraging producers and users to discuss the scope of 
the paper is not to ignore the danger that either might 
attempt, consciously or unconsciously, to predetermine or 
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alter the outcome of the estimate by posing the issues in ! 
terms of "loaded" questions. This appears to be a manageable f 
risk, and is likely to be outweighed by the benefits of a ! 
close and continuing exchange. The best insurance against f 
bias is probably a careful and continuing review of the I 
process by managers and participants. We found little, if t 
any, concern among producers or users with the possible I 
slanting of estimates as a result of a "cooking" process f 
involving the way the issues are posed. t 

The need for an estimate having been established and its ! 
scope worked out, the producer is next faced with a number of i 


decisions on procedural aspects of the process. These involve I 
such specific questions as how to handle the multidisciplinary I 
aspects of the paper, if any; the use of special methodologies, I 
especially those of a quantitative nature; and whether pro- f 
vision should be made for the presentation of competing views I 
on key issues, all matters on which critics have faulted the I 
producers of estimates. The succeeding pages discuss these t 
matters, along with other more general issues associated with 1 
the production process, including the use of net assessments, | 
coordination, and the handling of dissent. 


Multi discip l inary Work I 

There was general agreement among consumers of estimates i 

that the intelligence community has done far too few cross- I 

disciplinary or analytically -integrated studies in recent I 
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years. None of the dozen or so consumers with whom this 
specific subject was raised believed that estimates are now 
adequately interdisciplinary. Several thought that estimates 
were particularly poor in blending economic with political 
factors. Two felt that the intelligence community as a whole 
had done very little interdisciplinary work. One commented 
that NIE 35/36-1-76, The Middle East Military Balance, 1976 - 
1981, represented a "good try" in the multidisciplinary 
realm. This estimate attempted to interweave political, 
economic, and military factors into an appreciation of the 
relative strengths of various countries in the mid-East. 

Producers were less critical. Some maintained that esti- 
mates have contained interdisciplinary analysis all along, 
that consumers can be assured that estimates take account of 
all the main factors in the situation, even if some were left 
unstated. Several veteran intelligence officers believed 
that the ONE staff and Board had, in its heyday, constituted 
the multidisciplinary group par excellence . Other producers 
complained that the present product usually simply splices 
together separate sections of political, economic, or military 
analysis, leaving the reader to do his own synthesis of the 
relationships. It is the failure to produce this synthesis 
that spurs the criticism from the consumers. 

Almost everyone critical of the present interdisciplinary 
effort agreed that, while the community must do better, the 
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task will not be easy. There is, for one thing, little agree- 
ment yet on how best to accomplish good multidisciplinary 
synthesis or integration. Moreover, differing bureaucratic 
priorities and increasing specialization among analysts have, 

until recently at least, presented real problems in useful 

4 

interdisciplinary work. For another, the notion that multi- 
disciplinary analysis is "a good thing" has become perhaps 
overly fashionable in the intelligence community and outside 
it, and climbing recklessly on the bandwagon could easily 
lead to a drop in quality. This would be unfortunate, 
because the principle is sound and the need to apply it real. 

To produce a good interdisciplinary product, analysts 
and experts working on it must be organized in a way calcu- 
lated to bring their expertise to bear in a multidisciplinary 
manner. This probably necessitates placing the analysts in 
an environment with an effective task or project leader where 
meaty, wide-ranging discussion can take place prior to actual 
drafting so as to focus the drafting on the key questions from 
an interdisciplinary point of view at the outset. 

The discussion should be aimed at identifying the key 
issues and the cross-relationships among them, i.e., it should 
be a "synthesizing discussion." The objective is to produce 
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New organizational arrangements within the Directorate of 
Intelligence of CIA are intended in part to meet these problems. 
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in the drafting itself a narrative designed around the illumina- 
tion and analysis of the key issues, one that interweaves, 
according to their significance and interrelationships, political, 
economic, sociological, military and scientific factors bearing 
on the issue. It is this subtle, hard to produce synthesis 
that is the essence of a good interdisciplinary effort. 

New Analy tical Methodologies 

Another question is whether additional analytical leverage 
can be gained on estimative problems by reaching beyond tra- 
ditional techniques and utilizing quantitative, mathematical 
and systems-oriented methodologies. 

The alleged failure by producers to use new methodological 
tools of analysis has been cited by some critics as a serious 
flaw in the estimative process. Our interviews reflected 
little support for this charge. No consumer we talked with 
showed any deep-seated conviction that the intelligence com- 
munity was seriously remiss on this score. While virtually 
everyone agreed that the community ought to evaluate new 
analytical methods as they come along, no one argued that 
the product would be improved by their wholesale adoption, 
especially in estimative production, where short deadlines 
and the need for simplicity and relevance often work against 
complicated methodologies and presentations. 

We found a good deal of skepticism among consumers and 
producers about using computers and other "new” methodological 
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tools in preparing political or general estimates. Most 
described themselves as "willing to be persuaded," but as 
one producer put it, a long time is usually required before 
one can be sure that a new analytical technique will have an 
intelligence payoff, and this means that inevitably "you chase 
up some blind alleys." With limited analytical resources this 
can be an expensive proposition. 

Other producers pointed out that using new methodologies 
entails some risks. A strange new technical vocabulary- -a 
frequent by-product- -may well turn off the reader. Over- 
reliance on new techniques in writing estimates might imply 
greater precision in judgments than the facts warrant. And 
not all new analytical techniques - -not even those specifically 
designed for political analysis- -can be successfully adapted 
for estimative use. Many demand more information than analysts 
have or can get. Many are quantitative, and not all intelli- 
gence issues lend themselves to such treatment. 

Nevertheless, keeping up with the state of the art in 
analytical methodologies is an important obligation of all 
intelligence analysts, and several "new" techniques appear 
to promise at least limited applicability in estimating. 
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two possibilities; there are and will be others. The important 
thing is to strive to be open-minded and alert to possibilities 
as well as limitations. At this point, the new analytical 
methodologies, particularly the more radical and complicated 
ones, appear more suitable for conducting the basic analytical 
work supporting the production of estimates than for use in 
estimates themselves. 


Competing Analysis 


"...the issue of consensus versus competition 
in analysis represents a persistent conceptual 
dilemma for the intelligence community. Policy- 
makers tend to want one 'answer' to an intelligence 
question, but at the same time they do not want 

anything to be hidden from them Some members 

of the intelligence and foreign policy communities 
today argue that the consensus approach to intel- 
ligence production has improperly come to sub- 
stitute for competing centers of analysis which 
could deliver more and different interpretations 
on the critical questions on which only partial 
data is available." 


-- Final Report , Senate Select 
Committee on Intelligence 
Activities, 1976, Book 1, p. 272. 

The provision of competing centers of analysis is a con- 
cept that is getting considerable attention at present. The 
objective is to reduce the chances of overlooking or ignoring 
some vital aspect of an important analytical or estimative 
matter . 

In one sense, the principle is no stranger to the intel- 
ligence community. Currently, each separate intelligence 
component with competence in a particular field finds itself 
in competition with others to convince the audience that its 
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views are correct. Thus, each agency serves in a very real 
sense as a competing center of analysis. If this were not 
so, there would be no need for a coordination-dissent pro- 
cedure in the national estimative process. 

Some critics of the process argue, however, that it is 
not enough to have competition over ideas within and among 
the intelligence agencies themselves. They charge, in effect, 
that the intelligence community could become- -or has become- - 
a sort of closed corporation in which the free interplay of 
ideas and theories is no longer possible and important alter- 
native approaches are ignored. They allege that problems of 
"group think," of reinforcing consensus, and of mind set or 
bias often prevent the adequate exploration of analytical 
alternatives and the formulation and presentation of alter- 
native estimates. Critics cite the inability to provide an 
environment wherein analysts can easily depart, if they like, 
from the standard view on an issue as a main reason for the 
failures of intelligence to do such things as predict the 
outbreak of the Arab-Israeli war in 1973. 

We encountered only limited support for these assertions 
among consumers. Many of those commenting nevertheless made 
it clear that they wanted and expected all the informed views 
they could get, especially on the more important intelligence 
subjects. Some intelligence components do have a virtual 
monopoly of expertise on certain economic and scientific or 
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technical subjects. This makes for considerably less competit 
analysis on these subjects than many consumers - -or producers -- 
would like to see. 

In the effort to leave no analytical stones unturned, one 
relatively inexpensive approach is the use of a so-called 
"devil’s advocate." The concept provides that someone be made 
responsible for building a case opposed to that in a draft 
estimate, so that two sets of analyses and conclusions can 
be considered and weighed before the estimate is finally 
approved. The approach was tried during the production of 
NIE 11-3/8-75, Soviet Forces for Intercontinental Conflict 
T hrough the Mid-1980s , and we were told by some familiar with 
the exercise that it was very useful. Several of those 
interviewed favored using this process- -none was opposed- - 
but said that it should be employed only on the most impor- 
tant estimates. We agree. Clearly, there is a continuing 
need for efforts of this sort that force estimators to search 
out and seriously consider information and hypotheses that go 
against their own theories and views of reality. It would 
appear important, however, to confine use of the technique 
to important areas of estimates where there is substantial 
uncertainty and debate. It should not be used to help con- 
struct straw men. 

The only experiment involving a large scale effort to 
provide an alternative analytic approach was the so-called 
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"B-Team" exercise, used during the preparation of NIE 11-3/8-76, 
Soviet Forces for Intercontinental Conflict Through the Mid - 
1980s . The team was put together as a result of the dissatis- 
faction of certain members of the President's Foreign Intelli- 
gence Advisory Board with earlier estimates in this same annual 
series. Producers to whom we spoke before or during the experi- 
ment were generally neutral or slightly pessimistic in their 
views about its usefulness; consumers were guarded in their 
reactions. The widely publicized charges and counter-charges 
about the experiment, and the fact that it has been so recently 
completed, make it difficult at this point to evaluate its 
strengths and weaknesses or to offer an opinion about its 
overall value. Perhaps the only comment that can fairly be 
made now is that the idea of turning loose a fully staffed 
and funded "competing center of analysis" on an estimative 
problem is laudable in purpose, but likely to be expensive 
and difficult in execution. The competing teams must have 
the same data, the same rigorous analytical procedures, and 
the same degree of relentless objectivity if the exercise is 
to work at all. The hue and cry following the "B-Team" experi- 
ment suggests just how tricky a proposition this is likely to be. 

In all this, the problem is to encourage alternative 
analysis without artificiality and without a drop in the 
quality and coherence of the product, in a way that stimulates 
thinking rather than emotion, and within the constraints of 
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available time and resources. Ways must be found for encouraging 
the development and presentation of alternative views, and the 
time and means to give them a fair hearing and to incorporate 
them in the final text as appropriate. We suspect that the 
basic answer, as with so many other elements that make; for 
success in the business of estimating, lies in the skill, 
perceptiveness, and intellectual rigor of the estimative 
manager, which are probably more important to success than 
any organizational techniques that might be developed. This 
is not to say, however, that there may not be room for more 
formal arrangements, if the issue is important enough and if 
other factors seem to warrant such an approach. 

Net Assessments 

Although the use of net assessments is not frequently an 
issue in the production of estimates, it presents philosophi- 
cally troublesome problems and deserves discussion. 

The net assessment function, and the part that intelli- 
gence should play in it, are subjects of continuing confusion 
and controversy. Much of the confusion derives from the lack 
of a commonly accepted definition of a net assessment. The 
controversy is largely over where in the national security 
bureaucracy the function can most efficiently be performed, 
and with the least risk of having it distorted by policy biases. 

In the aspect that raises issues for intelligence, net 
assessment means pitting on paper, in carefully comparable 
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form, the strengths and weaknesses of an adversary power against 
those of the United States , in order to determine which is the 
stronger and in what respects. This is usually, though not 
necessarily, in terms of military forces. In the lingo of war 
gaming, a closely allied concept, net assessment poses "red" 
or enemy forces against "blue" or friendly forces. The rub 
is that those who do the assessing must have complete access 
to data on U.S. capabilities, present and prospective, as well 
as data on the capabilities of the adversary. Compiling the 
former, of course, lies well outside the intelligence com- 
munity's responsibilities and prerogatives. 

Of the producers and users of estimates with whom we 
discussed net assessments involving the U.S., none felt that 
the intelligence community should conduct them or include 
them in national estimates (although one saw the DCI as a 
possible producer, in some other context). To do so, they 
reasoned, would be to run an inordinate risk of transforming 
estimators into advocates or opponents of particular U.S. 
policies. They recognized that if net assessments are to 
be done, the intelligence community must furnish data and 
judgments on the capabilities of the adversary, but felt 
strongly that this should be the limit of the community's 
involvement. They believed that the overall responsibility 
for net assessments involving the U.S. must reside outside 
the community proper as it does at present. They did not 
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appear to rule out the presentation of comparative data on 
the U.S. and other countries in estimative products, or 
analyses involving comparisons, so long as the purpose and 
result was the elucidation of foreign capabilities, and not 
that of making net judgments. 

We find these views compelling. Getting involved on the 
U.S. side of a net assessment could rob the national estima- 
tive apparatus of one of its most important assets: its 
detachment from policy advocacy and from domestic disputes 
over the relative merits of proposed new weapons systems. 

With the U.S. intelligence effort already weakened by develop- 
ments in recent years, such involvement, apparent or real, 
could be disastrous. In making comparative analyses, which 
are quite another thing, estimators must preserve the dif- 
ficult but important distinction between that which helps 
the reader to better understand the capabilities of foreign 
powers, and that which becomes a net assessment in fact if 
not in name . 

Net assessments can involve comparisons between 
foreign countries only. In such cases, it is not only 
acceptable for estimates to include them but imperative, if 
they are to do their job properly. Several consumers told 
us that they would like to see more such net assessments, 
and we agree that it would be useful. Their preparation 
will probably be a growing assignment for the intelligence 
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community in the future, as power becomes more diffuse and 
as more blocs of states and regional complexes develop that 
have importance for U.S. national security. 


Coordination 


"...the process of coordinating a paper could 
be rewarding. The atmosphere becomes uncongenial 
to special pleading and to the urging of a paro- 
chial point of view by a particular agency But 

in addition to knocking down parochial prejudices, 
the process had a more positive aspect. It was 
a forum where people from all over town could 
exchange views, add to the store of community 
knowledge, and refine and sharpen their assess- 
ments of the course of events." 

--Sherman Kent, The Law and Custom 
of the National Intelligence 
Estimate , 1976, pp. 93-94. 

In the last sentence quoted above, Kent was speaking 
only of coordination. He might just as well have been dis- 
cussing the value of the whole estimative process from the 
point of view of the producers. The estimative process, 
including the drafting phase, is one of the few that requires 
a group of participants to work their way through an analytic 
problem, pulling together into a structured, reasoned whole 
what might otherwise remain as disparate strands of thought, 
impression, or opinion. In short, it gives the producers a 
base from which to compare, agree, and disagree. In the 
highly segmented world of most intelligence production today, 
with its frequent emphasis on quick reaction, such a result 
is important, regardless of the success of the product in 
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predicting the future or its immediate contribution to the 
policy process. 

That part of the estimative process that brings the experts 
on one subject together from separate agencies to scrutinize, 
debate, and revise the text of a draft estimate- - the so-called 
coordination process- -is in many ways the most important single 
step in the entire process. Often, it is also the most frus- 
trating. At its best, the interagency coordination meeting 
enriches and adds new substantive dimensions and insights to 
the draft. At its worst, it dissolves into interminable 
wrangling and stalemate, or even more unhappily, produces 
sterile language and flabby judgments in the name of consensus 
--and to get the messy business over with. Most intelligence 
officers who have participated more than casually in the estima- 
tive process will readily recall attending both types of session. 

Many users of estimates told us that it mattered little 
or not at all whether a paper was coordinated; for them, it 
was what the paper said that counted. If this attitude is 
prevalent, why bother with coordination at all, given the 
complexities of the process and the appreciable chance that 
it will have negative as well as positive effects? 

There are good reasons for doing so. The principal 
reason is that the coordination process goes to the heart 
of the concept of national intelligence. It allows the DCI, 
as the nation's chief intelligence officer, to express his 
views on the issues in a paper which is his and his alone, 
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while requiring other participants either to agree with him 
or to express their disagreements and the reasons for them. 
Perceived deficiencies in the process ought not to obscure 
this essential point. What it means, substantively, is that 
coordination brings to bear the talents of the entire intel- 
ligence community, provides a means of getting a variety of 
views into the paper, makes it more likely that the pertinent 
Issues will be covered, and, at its best, helps to define the 
issues and to sharpen them. Additionally, the give and take 
of coordination permit the participants to test their data, 
their assumptions and their hypotheses. It is thus incon- 
ceivable, despite the very real difficulties in producing 
coordinated assessments, that any top level decision maker 
could settle for less than a thorough interagency estimate 
on subjects of national importance. 

The pitfalls in the coordination process are never- 
theless numerous and important. Coordination can mask the 
very real and deeply-felt divisions that sometimes exist 
within the intelligence community on controversial issues. 

It can fuzz up clearly drawn, analytically sharp conclusions 
and projections, and cause an important point to be lost. 
Divergent views may be given short shrift by a "reinforcing 
consensus," and "submerged in a sea of conventional collec- 
tive wisdom." Haggling over major or minor issues can delay 
the completion of a paper well past the time when it would 
be most useful to its chief recipients. 
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Curing these problems is much harder than pointing them 
out, but a few general suggestions can be offered. One of the 
most important has to do with the official who serves as chair- 
man of an interagency coordination meeting. Whether the process 
proves product-enhancing or stultifying largely depends on this 
individual. To the extent that he or she is tough-minded, 
independent, and judicious, and is skilled in running a 
meeting and in separating serious substantive differences 
from trivial semantic hangups, the process is likely to be a 
fruitful one. 

It should also be borne in mind that a consensus is often 
not what the user of an estimate is looking for or needs. Con- 
sumers at all levels frequently told us that reaching a con- 
sensus was a part of the estimative process that ought to be 
played down. Several took pains to tell us that they were 
much more interested in seeing the final paper accurately 
reflect the range of views within the intelligence community 
on an important point. Some also wanted to get more of a 
flavor in reading the paper as to how the coordination process 
went: which issues elicited the most arguments among the 

interagency representatives, and which theories or lines of 
reasoning were advanced at the coordination table but dis- 
carded, and why? 

Another problem that bothered several interviewees was 
the internal difference of views that occasionally surfaced 
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on a subject among different components or officials of the 
same agency. This causes delays in the coordination process 
that would probably be reduced if participating agencies were 
to meet internally to sort themselves out on the issues before 
each round of interagency meetings took place. CIA, as the 
agency with prime responsibility for producing national intel- 
ligence has a special obligation to do this, but State, Defense 
and other members of the community should do more of it as well. 

Dissen t 

In the estimative process, coordination and dissent are 
two sides of the same coin. At their best, they help make an 
estimate more thoughtful and more useful to the policy maker 
than the product of a single agency. A well drafted dissent 
focuses a different view in such a way that the facts, assump- 
tions, and arguments employed are comparable with those for 
the position being dissented from. More fundamentally, the 
dissent mechanism is, as Sherman Kent puts it, "the indis- 
pensible corollary to the DCI's primacy" in the estimative 
field. If all estimates had to be negotiated to a compromise 
totally acceptable to all the participating agencies, the 
result would be pap; if the DCI were to insist that all 
agencies accept his views, the intelligence community would 
quickly disintegrate. 

The formal dissent is a device that has been used rather 
sparingly on most estimative subjects. One detailed review 
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of 83 estimates produced between 1951 and 1964 on a single 

5 

geographic region found only six formal dissents in the lot. 
That dissents have not been used more frequently may be partly 
due to an exaggerated notion that policy makers prefer single 
"bottom line" judgments. More recently, it has become clearer 
that the presentation of conflicting views on controversial 
or complex matters is not merely tolerable to the consumer, 
but that he actually prefers it. Several of those interviewed 
emphasized this preference. 

There is, of course, a world of difference between the 
value of a dissent taken to reflect an honest substantive 
difference with a predominant view and one taken primarily 
to support a bureaucratic position. There are many examples 
of both in the estimative record, and this seems bound to 
continue. But an increasing sophistication in elaborating 
dissents has characterized the process in the 1970s. To the 
degree that this is continued, it should be increasingly 
possible for the user to recognize readily--as many already 
do- -which dissents need to be taken seriously and which can 
be dismissed. 

One practice that should be pursued is the incorporation 
of dissents directly in the text instead of in footnotes. 
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The Arabs and the Estimates , ONE Staff Memorandum 58-65, 
4 Nov 65, TS. 
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The textual dissent is now frequently used, and most of the 
consumers who addressed this subject clearly prefer it. They 
feel that it provides a much better balance among competing 
ideas, and a better means of comparing them, than is possible 
if only one view is carried in the text and dissenting views 
are relegated to footnotes. 

Some consumers told us that they would like to see all 
the elements of the intelligence community holding a particular 
dissenting view clearly identified with the dissent. They 
also want to see an elaboration of the rationale behind the 
dissent. We agree that such refinements would further improve 
the mechanism of dissent and make estimates more complete and 
more useful. 

Summaries 

There was a strong feeling among the consumers inter- 
viewed that a good summary is an absolutely essential part 
of any estimate. Again and again, consumers emphasized that 
a tightly written summation that manages to convey accurately 
an estimate's key conclusions in comparatively brief form is 
the best possible device for ensuring high-level attention to 
the "message" of an estimate. Many added that lengthy esti- 
mates did not bother them per se --indeed, some users said 
they wanted and needed ample substantiating detail --so long 
as a crisp, well-written summary led off. 
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A very intensive, deliberate effort to draft such a 
summary needs to be untertaken for each estimate produced. 

Those responsible for reviewing the final draft of the paper 
should ensure that the summary is of the highest quality and 
accurately captures the flavor and tone of the estimate it- 
self. This is, of course, a tall order, but it is difficult 
to overstress its importance if estimates are to be well- 
received . 

Beyond matters of process affecting the substance of an 
estimate, the producers of an estimate have decisions to make 
bearing on the classification of the product, the manner in 
which it should be presented, and the possible scheduling of 
a so-called "post-mortem." The question may also arise as to 
whether an updated version of the product should be produced 
at periodic intervals. These matters are discussed below. 

Classification 

Some consumers were strongly of the view that estimates 
tend to be overclassified and thereby lose some of their 
potential audience and thus some of their utility. Both 
working-level users and officials high in the policy-making 
apparatus said, in effect: the lower the classification, 
the better. As in the past, highly classified material will 
have to be treated in estimative products. It seems only 
sensible, however, to adopt a general rule that highly classified 
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material, especially backup detail and supporting rationale, 
be relegated whenever possible to separate annexes or appen- 
dices. In this way, the basic estimate can be held to (or 
sanitized to) a lower classification, permitting the essentials 
to get across to a wider audience. 

We heard one other suggestion that struck us as useful 
in this regard: the periodic publication of brief abstracts 
of all the estimates written during a given period. If held 
to the Confidential or Secret level, such abstracts could 
receive wide dissemination, and apprise potential users (many 
of whom would have the necessary clearances) about documents 
of possible relevance to them. This could be particularly 
helpful to users of military/strategic or scientific/technical 
estimates, most of which require special handling or physical 
storage in remote vaulted areas. A publication of this sort 
might also aid in reducing the number of people who auto- 
matically get copies of published estimates. 


Written and Oral Presentation 


"Whatever the format and procedures, important 
intelligence should be presented in a way that can 
lead to discussion and questioning before decisions 
are made, so that the dangers of the policy maker 
misunderstanding the judgments (especially those 
expressed as probabilities) and the implications 
of such intelligence are reduced to a minimum." 

--William J. Barnds in the 
Report of the Commission on 
the Organization of the 
Government for the Conduct 
of Foreign Policy, June 1975, 
Volume 7, p. 32. 
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Although few would go so far as to agree with Marshall 
McLuhan that "the medium is the message," both consumers and 
producers alike seem strongly of the view that the presenta- 
tional medium chosen and the skill with which it is used are 
very important in getting across the message of an estimate. 
But, as with alternatives elsewhere in the estimative business, 
no one method of presentation pleases or meets the needs of 
everyone . 

The basic choices now available are essentially two: a 

6 

printed paper, or a "live" oral briefing. Each has certain 
rather obvious advantages and limitations. Most consumers -- 
seven of the ten who commented specifically on this point- - 
clearly preferred to get most or all their estimative material 
in written form. Of the remaining three, one felt that esti- 
mative conclusions should be offered in both written and oral 
form, one felt that oral briefings of estimates should be made 
available "more often," and one felt that the presentational 
format didn't matter much. All producers and former producers 
who commented said that they thought the intelligence community 
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Additional possibilities, including closed-circuit TV, 
video disk, graphic display of text or computer readouts are 
becoming increasingly feasible. In general, these seem better 
suited for use among specialists at the working level, than 
for communicating detailed judgments on complex subjects to 
top policy makers. 
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ought to explore more vigorously the feasibility of providing 
selected estimative conclusions orally to some consumers. 

We agree that the oral mode of communicating estimative 
material should be investigated, and used more often if there 
is sufficient consumer demand for it. Certainly, the pro- 
ducer of estimates should make available the services of a 
briefer as required. But oral briefings do not seem to offer 
any kind of panacea and, except for unusual circumstances, 
should supplement rather than replace a written text. Com- 
municating complex, sophisticated judgments and the supporting 
rationale orally is very impressive when it is done well, but 
it is difficult and subject to pitfalls, especially as regards 
nuances and qualifications, and it requires special skills that 
may not always be available. Even when they are, the intended 
audience may be too busy to give the briefer the time or 
attention needed to do an effective job. Our view is that 
the brunt of any effort to improve presentational formats 
ought to go into making written estimates as succinct, 
readable, and responsive to different levels of need as 
possible. Efforts should also be made to produce, to the 
degree that resources permit, several versions of key esti- 
mates for consumers with need for different levels of detail. 

"Post-Mo rte ms" 

Over the years, the term "post-mortem" has been used in 
the intelligence community to describe two quite different 
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processes. It may mean the formal qualitative review of an 
estimate or a series of estimates some time after publication 
to judge the accuracy of the predictions and the soundness of 
the analysis. It may also mean the review of an estimate soon 
after publication to investigate the adequacy of the informa- 
tion on which it was based and to identify the need for future 
collection or analysis to fill the gaps. 

Post-mortems of the latter type were once quite common, 
but were later abandoned, because they became highly pro- 
forma and routinized, and failed to accomplish their purpose. 

The only recent example of which we are aware was conducted 
on NIE 11-3/8-75, Soviet Forces for Intercontinental Conflict 
Th rough th e Mid-1980s . It was carried out as an experiment 
by the various departmental representatives who had just com- 
pleted the estimate in order to identify gaps in intelligence 
and areas of special concern that required more attention next 
time around. The participants were pleased with the experiment. 

We agree with those holding that the device can be help- 
ful if used sparingly and on estimates of prime importance. 

We think a post-mortem should be undertaken shortly after an 
estimate is completed, should involve both producers and con- 
sumers, and should be focused on identifying substantive gaps 
and problem areas rather than on Monday-morning-quarterbacking 
the estimate's judgments and predictions. The role we see 
formal post-mortems performing best is that of helping close 
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the circle in the intelligence production process. By pointing 
up substantive weaknesses and missing pieces of the puzzle, the 
post-mortem could become a valuable tool in helping set realistic 
priorities and requirements as guides for intelligence collection 
efforts . 

One useful variant would not await the completion of an 
estimate, but would be carried on concurrently with the drafting 
as an integral part of the process. There is much to the propo- 
sition that the most efficacious time to identify deficiencies 
in information and analysis, and to initiate corrective action, 
is at the time the analytical process is concentrated on a 
substantive matter during the process of drafting. 

Post-mortems on the quality of estimates have been rela- 
tively rare, particularly in recent years. Most recent post- 
mortems of this kind have been in response to "intelligence 
failures." They have encompassed not just estimates, but 
the entire range of intelligence coverage on a subject, from 
special memoranda to current intelligence reporting, situation 
reports and even individual, raw intelligence reports. The 
post-mortems on the October 1973 Arab-Israeli war and on the 
Cyprus crisis in 1974 are examples. The producers who com- 
mented on post-mortems concerning quality favored them as 
helpful if they were used only on estimates of major impor- 
tance and if they included provision for feedback from 
consumers . 
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Post-Publication Update 

We encountered almost no support for the notion that 
estimates ought to be revised and reissued at fixed intervals, 
except in the case of the annual Soviet strategic estimate, 

NIE 11-3/8, on which there was substantial user demand for a 
new version each year. One Department of State country director 
did express an interest in an annual estimate on his country, 
even if nothing significant had occurred in the interim. But 
his country is a "denied area" as far as reporting channels 
are concerned, and he was the only user who took this position. 
Our own view is that estimates should be updated, as a rule, 
only when a significant change in the situation or in the 
questions to be addressed requires it. 

The problem is that there is at present no systematic 
method for determining whether such changes have occurred. 

In our view, all estimates should be reviewed periodically 
to decide whether an update is required. Such a review would 
involve a sort of post-mortem to determine whether the existing 
estimate is deficient in scope, analysis or in the pertinence 
and accuracy of its judgments. Systematic stock-taking would 
not only help insure that particular estimates remain up-to- 
date, but might also provide more general insights into past 
mistakes which can be minimized or avoided in the future. 
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IV 

ORGANIZATION FOR THE PRODUCTION OF ESTIMATES 

The ultimate quality and utility of an estimate depend 
primarily on the personal skills and insights of those who 
produce it. But their ability to use these qualities effec- 
tively is determined in no small measure by the organizational 
environment in which they work. The present chapter focuses 
on certain aspects of that environment. 

The question of how the machinery for producing estimates 
ought to be structured was among the topics most frequently 
raised in our interviews. Producers, not surprisingly, had 
stronger views than consumers. While we encountered a wide 
variety of views, most were variations of two basic proposals: 
revert to the former ONE system or something like it, or 
retain and perhaps modify the current NIO system. 

Under the NIO system, the estimative process can begin, 
as it has all along, with a request from a consumer. The 
appropriate NIO makes a recommendation to the DCI, who 
approves or disapproves. If he approves, the NIO becomes 
responsible for the proposed estimate, the choice of NIO 
depending on the subject matter involved. The NIO can also 
initiate a proposal, subject to the approval of the DCI, and 
the DCI himself may do so. The NIO may or may not choose to 
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convene a meeting to decide on terms of reference or to receive 
oral contributions, but almost always does so. Such meetings 
involve representatives of the agencies and departments of the 
intelligence community most interested in the completed estimate 
and best able to contribute to it, and sometimes include policy 
makers to help assure that the proper questions are addressed. 

The NIO selects a drafter (or drafting team) from among 
the same agencies, in consultation with the parent agency or 
component, and research and writing get under way. The 
drafters work under the guidance of the NIO; written con- 
tributions may or may not be requested from the participating 
agencies. The draft, when completed, is circulated to the 
participating components for review prior to its consideration 
and coordination at an interagency meeting. This meeting, of 
representatives of the intelligence chiefs of the participating 
components, is usually chaired by the NIO, but on occasion he 
designates someone else. When coordination is completed, the 
draft is reviewed by the DCI (who may also have involved him- 
self at earlier stages) and considered by the National Foreign 
Intelligence Board. Once approved by the DCI, with the agree- 
ment or disagreement of other NFIB principals shown, the esti- 
mate is published and disseminated. 

The system in the former Office of National Estimates 
differed in important respects. Except for three major 
military estimates (which were drafted by CIA teams) , drafts 
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were prepared by a small staff of analysts in ONE, who devoted 
their full time to such work. These drafts were reviewed by 
the Board of National Estimates, a small group of distinguished 
generalists responsible to the DCI. The Board provided a col- 
legial review, designed to ensure that the proper questions had 
been asked and answered, that there were no gaps in the treat- 
ment of the subject, and to test and if appropriate to challenge 
the judgments in the draft. Drafts were then sent for review 
to the participating agencies. Subsequently, representatives 
of the chiefs of these agencies met with the Board to comment 
on the draft and to coordinate it, prior to its submission to 
the United States Intelligence Board (now NFIB) . 

There is still another difference in the two systems. 

The staff and Board of ONE typically spent far less time on 
liaison with consumers than do most NIOs, who are explicitly 
charged with making such liaison an important part of their 
duties. This lesser emphasis was in part to ensure that the 
separation between intelligence and policy was maintained, a 
separation regarded at that time as highly important by many 
in the intelligence community. 

The current system gives each individual NIO a larger 
measure of autonomy and responsibility for the production of 
estimates than any one individual had under the ONE system. 

It also encourages and requires a close working relationship 
between each NIO and the main producers and consumers in his 
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particular area. The former results in more structural flexi- 
bility and responsiveness than was possible under the ONE sys- 
tem. The latter makes possible better communication between 
producers and consumers. Both results were highly valued by 
most of the consumers who commented on the subject. A third 
major benefit, in the eyes of some producers, is that the NIO 
system puts the drafting responsibility on analysts who deal 
with a subject on a day-to-day basis, and who, theoretically 
at least, have the greatest expertise on that subject. 

At the same time, many producers and some consumers were 
quite vocal about what they saw as shortcomings in the NIO 
system. One frequently heard criticism was that estimates 
before 1973 were of more uniformly high quality than those 
produced now. Another was of the ad hoc nature of the 
drafting process, whereby analysts with little or no expe- 
rience in drafting estimates are asked to do the job. It 
was pointed out that the best substantive expert in town is 
not necessarily able to write a good estimate, and that even 
if he is, he may be so busy responding to a host of demands 
on his time that he is unavailable to the NIO or forced to 
divide his time among different tasks and slight his drafting 
assignment . 

The system, as seen by some, also places an extraordinarily 
heavy burden on one person--the NIO. This individual must take 
on almost single-handedly not only the substantive responsibility 
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for the accuracy and thoroughness of the estimate but also 
the managerial and diplomatic chores involved in steering it 
through the coordination process. At the same time, he must 
cope with the numerous other responsibilities of his position. 
There is no collegial review apparatus to spot weak or missing 
links in the paper, or to challenge the conventional wisdom, 
as was the case under the ONE system. Finally, a single NIO 
is probably less able than a group to deflect advocates of 
particular bureaucratic viewpoints. 

Efforts have been made to respond to some of these prob- 
lems. An Estimates Advisory Panel made up of distinguished 
individuals from outside the intelligence community has been 
approved though not yet activated; it would supply an element 
of review. The recent reorganization of the production offices 
in CIA's Directorate of Intelligence was undertaken in part to 
put more analytical strength into the Agency's estimative 
effort and to make it bureaucratically easier for substantive 
experts to be freed from competing demands on their time while 
working on estimates. 

We seriously question, however, whether these steps, 
laudable though they may be, go far enough. If the President, 
his top advisors in the national security and foreign affairs 
field and the American public are to be confident that esti- 
mates contain the very best intelligence judgments that can 
be made, further organizational improvements need to be 
undertaken. 
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Internal Collegial Review 

A small group of distinguished generalists of proven 
wisdom and ability, chosen from both within and outside the 
intelligence community, ought to be reinserted into the 
estimative process as a collegial review body. 

In making this recommendation, we are aware of the sharply 
contrasting views that have been expressed about the work of 
the former Board of National Estimates, which performed the 
collegial review function until it was abolished in 1973. 

Its detractors charged that the institution was out of touch 
with the needs of customers and that it encouraged an undue 
search for consensus at the expense of sharpness in estima- 
tive language and judgments. Its defenders maintained that 
its labors resulted in a more thoughtful and sophisticated 
product, and that it helped guarantee thoroughness and 
objectivity of analysis and presentation. At any rate, 
since the demise of the Board there has been no collegial 
review of estimates, save the necessarily brief and often 
cursory one that occurs when the National Foreign Intelligence 
Board meets to consider a draft. 

Arguments over the value of a collegial review of esti- 
mates by a body of generalists continue to this day. Most 
of the consumers whom we interviewed and who commented on 
the subject clearly supported the concept. Producers were 
mixed in their reactions. Some of those at or near the 
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working level tended almost instinctively to oppose it. As 

one put it, in extreme terms, "the less review the better; 

the customers' reaction is the only review we need." There ! 

tended to be greater support for the concept among senior i 

intelligence officers, present and past, many of whom felt : 

that collegial review by generalists is a vital part of the j 

process that should not be neglected any longer. 

We find the arguments of those favoring collegial review ! 

persuasive. Better quality could ensue from the review of a 
draft by a group of thoughtful, analytically incisive indi- 
viduals who are sharp enough to catch faulty logic, wishful ! 

thinking, and inconsistencies. More thoroughness could ! 

result from the examination of a draft estimate by a group 
of generalists, each bringing to bear his own approach and 
knowledge, so that the draft is considered from a variety I 

of angles. Greater objectivity could be attained because a 5 

group independent of any bureaucratic loyalties would be I 

better able than a single individual to fend off pressures i 

to alter judgments or avoid hard calls. Also, generalists 
can call attention to some important element in a situation 
that the experts are apt to take for granted and say nothing 
about, or to facets of a subject that they might have overlooked. 

Members of the collegial review body ought to be drawn i 

from various professions in and out of government, with the 
widest possible representation from the major academic 
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disciplines used in the estimative process. Inclusion of a 
political economist, a scientist, an economist, a military 
specialist and a sociologist or anthropologist might be a 
good start. Members should be appointed by the DCI, and be 
responsible only to him. To prevent bureaucratic hardening 
of the arteries, members should ordinarily not serve for 
more than four or five years. The review body should be a 
full time operation, and should be close to the DCI both 
organizationally and physically. Beside examining estimative 
drafts, the group might generate or discuss ideas for new 
estimates, review old ones to see if they need to be updated, 
and perhaps stimulate and comment on thoughtful single agency 
products inappropriate for national estimative treatment. 

Review by Outside Consultants 

The Estimates Advisory Panel (EAP) could, if implemented, 
serve as an adjunct to the collegial review body, and possibly 
as an eventual source of recruits for it. 

As originally conceived, the EAP was to consist of about 
three dozen carefully selected experts, primarily if not 
entirely from outside the intelligence community. Members 
would be invited by NIOs to gather in small sub-panels to 
review specific estimative projects while they were under 
way and to provide comments and recommendations. 

The use of outside consultants on estimative problems 
is not new, dating back to the beginnings of national estimates 
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in 1950. Those familiar with the history of their use felt, 
for the most part, that only indifferent success had been 
achieved overall, but that sometimes the results were useful 
Intelligence producers who commented on the EAP were only 
mildly optimistic. They pointed out that its real value 
would be directly proportionate to the caliber of the people 
enlisted, and expressed skepticism that enough able people 
could spare the time to make the idea work well. 

Their caution seems well founded, for the problems in 
using outside consultants are substantial. A consultant 
must be willing and able to spend a good deal of time going 
over the intelligence data on which an estimate is based; 
this can be a real problem for busy, prominent specialists 
with their own careers and interests. Another problem is 
that the consultant may find it difficult to cope with the 
rather special world in which intelligence producers operate 
As one senior official said, "if it is hard for a producer 
to get into the mind of a policy maker, it will be three 
times as hard for a consultant to do it." 

Nevertheless, the EAP concept seems worth a try. Con- 
sultants have performed a useful role in the estimative 
process in the past, and can do so again. At the least, 
assuming that top notch people are recruited and used, the 
EAP could help allay concerns that national estimates are 
not the result of the best thinking that can be obtained on 
a subject. 
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Changes in Current Practices 

There remains the question of what changes should be 
made in the basic practices now employed for producing 
national estimates. We recommend two. 

The proposal for inserting a collegial review body into 
the process, as recommended above, could or could not consti- 
tute a basic change, depending on the way it is implemented. 

On the one hand, a collegial review group could serve in an 
advisory role to the NIOs, with the latter free to accept or 
reject its advice and recommendations. On the other hand, 
acting for the DCI, it could have the last word during the 
production process on matters of scope and content, much as 
the former Board of National Estimates did. The ultimate 
responsibility and authority for the scope and content of 
national estimates would of course be the DCI's, as it has 
been all along. 

In making our recommendation for a review group, we had 
in mind the latter alternative, for a number of reasons. If 
the review group were only advisory, it would be no more than 
a group of internal consultants. We doubt that it could then 
be justified, particularly as a full time body, and it might 
have difficulty in acquiring and holding the caliber of people 
required. Substantively, it would not have the authority to 
go along with its responsibility for insuring, through a 
collegial approach, high quality and the thorough and com- 
plete treatment of a subject. 
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A collegial review group with authority would serve two 
other purposes. It would be an additional means of insuring 
the objectivity and disinterestedness of national estimates, 
qualities that should be among their prime contributions to 
the policy-making process. This contribution is an important 
one, for the comparative advantage of intelligence officers 
vis-a-vis their policy making counterparts lies not in their 
being more able, better informed, or better intentioned, but 
in the vantage point from which they do their work and in 
their separation from the policy process. Second, a review 
body with authority over substance would serve as an additional 
means of both highlighting and exercising the primacy of the 
DCI in the production of national estimative intelligence. 

We urge a second basic change in current procedures, 
which we consider to be as important as the first. That is 
the establishment of a small staff of analysts who would be 
charged with the initial drafting of estimates. It would 
have no other duties. The staff, like the collegial review 
group, would be an instrumentality of the DCI. Care should 
be taken to insure both the substance and appearance of this 
relationship, and the separateness of the staff from the 
constituent elements of the intelligence community, for the 
reasons already suggested in the preceding paragraph. 

There are good reasons for having a full time staff to 
draft estimates. The drafting of estimates requires particular 
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skills, which are not easily found or developed. Native 
ability and substantive expertise do not guarantee them, for 
they involve the crafting of a distinctive art form- -the 
national estimate- -and the development of perspectives and 
approaches suitable for it. Experience strongly suggests 
that these skills are best developed through specialization 
and experience, i.e., by the repeated exposure of an indi- 
vidual to the actual task of drafting. The present method 
of drafting has created some problems, as mentioned earlier, 
and almost certainly accounts for much of the "unevenness" 
in the product frequently mentioned to us. 

We recognize that it is untidy to make the above two 
suggestions for change in the present system of producing 
estimates, and to go no further. It seemed better, however, 
to confine ourselves to a discussion of principles, rather 
than to try to set forth a detailed blueprint of what the 
estimative machinery might look like after the changes were 
made. In the same vein, we make an additional recommendation. 
According to the comments of those we interviewed, and by our 
own reckoning, the NIOs have made a most valuable contribution 
in carrying out their explicit responsibility for maintaining 
contact with the principal users of estimates. This has 
helped to make estimates more relevant and responsive to 
their needs. We urge that provision for this kind of 
dialogue be maintained. 
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V 

THE USER’S ROLE IN ESTIMATES 

This study has stressed that if estimates are to be 
relevant and useful, producers must have a good understanding 
of the needs of users. Common sense suggests that such under- 
standing can best be acquired if producers and users are 
reasonably closely linked, at least to the extent of com- 
municating directly on matters of scope, timing and the 
issues to be addressed in an estimate. On this basic pre- 
cept there is no real dispute, but there are differences of 
opinion on how to define "reasonably closely" and on whether 
the benefits of close association outweigh the risks. 

A number of policy makers saw little need for any 
additional effort to provide guidance to the producers of 
estimative intelligence. Some thought that if an intelli- 
gence producer were competent and well informed, he could 
determine on his own what users needed and wanted. Most 
policy makers, however, considered that efforts to improve 
communication were necessary and desirable. Generally, they 
felt that the NIOs had done much to bridge the gap between 
policy officers and the producers of intelligence, but that 
much remains to be done. 

If there is to he an effective dialogue between the 
producers and users of estimates, it would seem necessary 
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that certain conditions be met. The most important, noted 
before in this study, is clear evidence of interest on the 
part of the President and other senior policy officials in 
the utilization of estimative intelligence as part of the 
policy-making process. The President in particular, as the 
chief "user" of intelligence, sets the tone. Second, senior 
officials on both the intelligence and policy side must 
actively foster the dialogue and devote part of their time 
to producer-user relationships; the key individuals are the 
DC I and the President's National Security Advisor. To date, 
these conditions have been satisfied only partially and 
sporadically . 

In November, 1971, the National Security Council Intel- 
ligence Committee (NSCIC) was established to provide user 
"direction and guidance on national intelligence needs," and 

"evaluation of intelligence products from the viewpoint of 
7 

the user." It was made up of senior policy makers, plus 
the DCI . Largely because of a lack of top-level attention, 
the NSCIC languished, holding only one substantive meeting 
during its entire five-year existence. Its second Working 
Group, established in late 1974, made only a small start in 
educating users about the products and capabilities of the 
intelligence community. The NSCIC was criticized for its 


7 

White House press release, November 5, 1971. 
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inactivity by both the Murphy Commission and the Senate Select 
Committee on Intelligence. 

By the time the Select Committee submitted its report, 
the NSCIC had been abolished. Executive Order 11905 of 
February 18, 1976, requires the DCI to consult with users 
of intelligence to ensure the "timeliness, relevancy, and 
quality of the intelligence product" and to "establish such 
committees of collectors, producers, and users of intelligence 
to assist in his conduct of his responsibilities as he deems 
appropriate." In implementation of this order, the DCI 
charged the National Intelligence Officers with determining 
consumer needs for intelligence and gave his Deputy for the 
Intelligence Community the responsibility for the evaluation 
of intelligence products. At present, there is no formal 
organization of users to provide guidance to the producers 
of intelligence. It is probably fair to say that the col- 
lection and production of intelligence are governed more by 
what producers think is needed than by guidance from users. 

C 1 ose User-P r oducer Links: Risks and Bene fits 

There are two basic schools of thought about the proper 
relationship between intelligence producers and policy makers. 
The first holds that intelligence analysts ought to maintain 
a certain remoteness from those involved in decision making 
"so as to keep intelligence pure and untainted by policy 
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8 

pressures," as Hanson Baldwin put it. The other asserts that 
close cooperation between intelligence and policy officers is 
not only permissible but mandatory, if both are to do their 
jobs properly. Most of the consumers and producers we inter- 
viewed held the latter view. They felt that the risks of a 
close relationship, although not to be taken lightly, are 
nevertheless tolerable in view of the potential benefits to 
both sides. 

The chief concern of those who worry most about the risks 
is that a close association will in time make the estimators 
too sensitive to the user’s interests and attitudes. As one 
former senior intelligence officer put it, there is a distinct 
danger that the producer will get so wrapped up in day-to- 
day support of the policy maker that he will become, con- 
sciously or unconsciously, a member of the policy team. By 
so doing, he risks losing his objectivity and his ability 
to step back, take the longer view, and go beyond the opera- 
tional considerations of the moment to product a paper that 
provides longer range perspectives and perhaps challenges 
the postulates on which policy is based. For their part, 
policy makers can legitimately be concerned that intelligence 
producers will intrude into their territory if the relationship 


8 — — 

Hanson Baldwin, "The Future of Intelligence," Strategic 
Review , Summer, 1976. 
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with them becomes quite close- - though no policy makers that 
we talked to complained of this having happened. 

The primary benefit of a close working relationship, as 
we and our respondents see it, is a clearer, more realistic 
appreciation by each side of the capabilities, limitations 
and needs of the other. The producer of intelligence would 
have less need to guess about what is wanted by the users of 
his products. He could acquire a better appreciation of the 
problems preoccupying the policy maker, and what issues are 
paramount in his thinking. As one of our respondents put 
it, it is necessary for the producer of estimative intelli- 
gence to get inside the mind of the user; this cannot be done 
without close and continuing association. Such association 
would facilitate the production of papers which are relevant 
to the needs of the policy maker, and responsive to them. 

For their part, many policy makers have an imperfect 
view of what the intelligence business is all about. More 
particularly, they are unaware of what estimates can and 
cannot be expected to do for them, and of the difficulties 
that are sometimes involved in zeroing in from a distance 
on matters of interest to them. There also appears to be an 
insufficient understanding of the process of producing esti- 
mates and of the nature of the product that results, with the 
consequence that an estimate is viewed by some as no different 
from any other intelligence product. Perceived deficiencies 
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in the product have contributed to this viewpoint. So has the 
failure to understand that a national estimate is the voice of 
the DC I , that coordination has many positive benefits, and 
that the vantage point of the intelligence estimator permits 
him to make a distinct, if not unique, contribution. 

Providing for Closer Contacts 

We found a general feeling among both users and producers 
that providing for the right kind of communication between 
them will not be easy. One very obvious constraint on both 
sides, but especially on policy makers, is a lack of time. 

Policy people who admit to neglecting their relations with 
intelligence producers complain that there simply is not 
enough time in their operational and crisis-oriented schedules 
to fit in regular meetings with intelligence analysts, no matter 
how desirable this might be. Additionally, the fact that users 
and producers are physically located up to an hour apart is a 
strong deterrent to impromptu meetings when time is available. 

A solution sometimes suggested for bringing intelligence 
and policy people together more regularly is the creation of 
an organization like the NSCIC, but without that group's 
liabilities. Among the few producers and users who commented 
there was skepticism, however, about the efficacy of a new 
attempt to provide a formal, institutional solution. Such 
skepticism is understandable in the light of experience. 

Unless vigorous and continuing leadership is exerted from 
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the top, a new organization might soon atrophy. As the pres- 
sures of more immediate operational concerns grew, top-echelon 
members of the group would be increasingly tempted to designate 
lower ranking officers to attend meetings. This would reduce 
the organization's power to make decisions and to provide 
guidance . 

This alone is not reason enough to shy away from a new 
effort to establish such a group, and in fact such an effort 
ought to be made. Even a lively second or third echelon users' 
organization would provide a forum for airing the concerns and 
needs of policy makers. It would be wise, however, not to 
expect too much of such a group, given the problems and 
pressures described above. 

Among the subjects it might tackle are the following. 

On estimates: 

--topics for national intelligence estimates; 

--terms of reference; 

- -priorities ; 

--the evaluation of published estimates; and, 

--the role of estimates in support of "opera- 
tional" requirements and in crisis situations. 

More generally: 

--intelligence research programs; 

--significant gaps in intelligence coverage; 

--the allocation of intelligence resources as 
it affects the final product; 
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--conflicting demands on intelligence; 

--imbalances in the intelligence effort; and, 

--the broad capabilities, limitations and 

problems of intelligence collection, analysis 
and production, as they affect the ability 
of intelligence to support policy makers. 

Even if a formal group is established, it seems evident 
that the improvement of communication between policy officers 
and intelligence officers will depend mostly, as it has all 
along, on the initiatives taken by individuals to get to know 
one another better. Furthermore, it is clear that most of the 
burden will be on the producers of intelligence. As one quite 
senior State Department official put it: "the policy maker 
does not communicate with intelligence- -you know that. The 
intelligence officer has to anticipate and do everthing. 

The policy man will not tell him enough, early enough." 

Another senior customer said that he found it difficult to 
provide advance guidance, and that intelligence should not 
look for it. He added that the intelligence producer should 
instead derive guidance from a greater interaction with users, 
so that he could identify independently what was needed. 

Certain related themes emerged from our interviews. 
Several policy makers stressed the importance of the per- 
sonal relationship between producers and users. That is, 
only if there were relations of mutual confidence would a 
policy maker be prepared to express his innermost thoughts 
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or to welcome periodic get-togethers at which perceptions 
could be compared. There were also suggestions that the 
intelligence community do market research, in a rather 
formal way. As one senior policy maker put it, intelligence 
has to find ways of getting its message across, and this is 
a marketing problem. 

Within the framework of an expanded dialogue, each side 
needs to work harder on certain aspects of the relationship. 

The user, busy though he is, should do all he can to ensure 
that the intelligence producer is aware of trends in policy 
that may create new requirements for estimates. He also 
ought to strive to find out more about what intelligence 
can, and cannot, do for him. This would involve a far 
better understanding than many policy makers now have of 
just how the intelligence community functions. 

Producers have a responsibility for seeing that policy 
makers acquire such an understanding. The main users of 
estimates are busy people. Moreover, there is fairly rapid 
turnover in many of the key positions in the national security 
and foreign policy field. Under these circumstances, the 
intelligence community has probably been remiss in not having 
developed a program for acquainting its customers with the 
capabilities and limitations of estimative intelligence. 

When this is done now, it is on an ad hoc basis, with little 
consistency or continuity. As simple a thing as the production 
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of a package of sophisticated briefing material would probably 
help, especially if it were delivered personally by an intel- 
ligence officer with whom the user had or was developing a 
working relationship. 

Producers also owe it to themselves, and to the ultimate 
utility of their product, to work harder than many now do to 
absorb what is readily knowable about current policies and 
the preoccupations of policy makers. An alert intelligence 
officer can acquire a substantial understanding of at least 
the thrust and direction of U.S. policy in his area simply 
by keeping his eyes and ears open. Several producers 
admitted as much, and ventured that this was often enough 
to permit them to do their jobs. And, of course, it is the 
duty of the producer of estimates not only to tell the con- 
sumer what he wants to know, but also what the producer 
thinks the consumer ought to know. 

This gives rise to two other thoughts on this subject. 

The first is that those charged with producing estimates can 
learn much about the concerns of policy makers, particularly 
those at the Department of State, from their colleagues in 
the Directorate of Operations (DDO) . Officers of the DDO 
maintain continuing contact with their opposite numbers at 

the Department . 

Also, DDO division chiefs meet regularly with the Assistant 
Secretaries for their areas. At a minimum, the results of 
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such meetings ought to be passed on to the producers of 
estimates; it would be even better if the latter were invited 
to participate as appropriate. 

Second, one of the best ways for intelligence producers 
to get to know the needs and concerns of users is to work in 
or close to a policy position for a time. We found that the 
most perceptive and pertinent comments on user-producer rela- 
tions during our interviews came from the relatively few 
individuals who served in both types of position. But such 
cross-fertilization should be practiced carefully to avoid 
conflicts of interest and the blurring of the lines of separa- 
tion that both users and producers believe it essential to 
maintain. There are also serious impediments to assigning 
policy officers temporarily to intelligence roles. For care- 
fully selected intelligence officers, however, a stint in a 
policy office could be highly useful as a device to foster 
better mutual understanding. 
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VI 

THE FUTURE ROLE OF ESTIMATES 


What we have heard and read during the course of this 

study convinces us that national intelligence estimates will 

have a useful role to play in the policy-making process for 

the foreseeable future. Whether they actually play such a 

role will be largely up to the intelligence community itself. 

Far from having outlived their usefulness, estimates would 

seem to be more urgently required than ever. There is good 

reason to think this is true, to one degree or another, of 

9 

most kinds of finished intelligence. But because estimates 
alone attempt to discern what lies ahead, they surely should 
get high priority in terms of our total intelligence effort. 

Other reasons for their potential importance are not 
hard to adduce. The world with which intelligence and policy 
must deal is still changing with breathtaking speed. Rela- 
tions between and among nations and people are becoming more 
and more complex. As they do, it is getting steadily more 
difficult for the policy maker, pressed by a great many more 
immediate concerns, to find the time to take the long view. 


See The Future Market f or Finis he d Intelligence , (S) 
a CSI study" issued in August' llTTFj - ’ for a more detailed look 
at finished intelligence needs in the years to come. 
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Nor can he usually afford to sort out for himself the many 
interacting factors that will affect the prospective behavior 
of a foreign entity. And unless he is directly involved in 
such matters, he will rarely be able to stay abreast of new 
and impending scientific or technological changes that may 
impact upon his responsibilities. 

This is where intelligence comes in- -or should, because 
these are the sorts of things that useful estimates come to 
grips with for their readers. Estimates have done this well 
at times and poorly at others, but will need to be much 
more effective and consistent from now on. The task will not 
be easy. For one thing, the growing complexity and inter- 
relatedness of the world's military, technological, economic, 
political, and social affairs will make it much more difficult 
to be clear and precise, or to phrase estimative judgments in 
simple declarative sentences. More and more tangible and 
intangible factors will have to be considered before reasonable 
conclusions can be formed. This will make it increasingly 
important to weigh carefully all the alternative views on 
crucial issues and to articulate them clearly. It also 
means that the underlying data base is likely to become 
larger, or clouded by conflicting information. Yet to 
ignore or pass lightly over parts of the data would be risky. 

For another, it is likely that the resources of the 
intelligence community, like those of the nation as a whole, 
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will continue to be stretched to accommodate other urgent 
needs. This may well mean that estimates will have to be 
produced more economically, in a shorter time span, and for 
a wider audience. Eventually this seems bound to entail pro- 
ducing fewer estimates. Producers and users will no longer 
be able to enjoy the luxury of trying to know everything 
about everybody, but will have to concentrate on those 
countries and problems that affect our security the most. 

This means a greater willingness to say "no" to requests 
for estimates on less essential matters. 

"Quality" should become the watchword, because the need 
for estimative excellence will continue to grow as U.S. power 
and self-sufficiency become more circumscribed. Our national 
well-being will depend more heavily on knowing accurately 
what our adversaries and allies may be up to. This does not 
necessarily mean refining the ability to predict specific 
events. It will, rather, mean being close to the mark in 
forecasting trends, measuring capabilities of important foes 
and friends, and assessing their intentions. 

Even when quality must be cut back, the plate will remain 
full. Surely there will be no scarcity of important estima- 
tive issues ahead, including some that are critical and a 
few that are explosive. Unforeseeable crises will occur and 
troublesome developments are bound to arise. Many topics of 
present concern will remain and become even more difficult 
to handle, 
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Assuming that well thought-out, relevant estimates are 
produced, there need be few worries about a market. The most 
important customers will continue to be the President and his 
chief national security advisors. To the degree that these 
leaders find estimates useful, and to the extent that esti- 
mates have an established place, formal or informal, in the 
policy-making structure, their market will be assured. The 
frequent turnover of military and political decision makers 
at intermediate and lower levels is a further guarantee that 
a potential audience for estimates will always exist. The 
DCI , however, will continue to be the key person in getting 
estimates read and taken seriously. If the DCI is convinced 
that estimates are as good as they can be and tries to con- 
vince others, estimates should have few problems in reaching 
the right people. 

And the "right people" may well turn out to include a 
much larger number in the future than it has up to now. 
Congressional interest in the conduct of foreign and national 
security policy and the new emphasis on performing govern- 
mental functions as openly as possible suggest that estimates 
will increasingly need to be made available to the Congress, 
and perhaps even to the public in some form. To do so will 
require a good deal of careful work in the area of determining 
what must remain classified in order to protect intelligence 
sources and methods and to avoid damaging our relations with 
friendly governments. 
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In conclusion: their place in the policy-making process 
may remain modest and their ability to provide satisfactory 
answers about the shape of the future will be limited, but 
for all that, national intelligence estimates will be needed 
more than ever. There should be a sustained effort to do 
them as well as possible. 
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APPENDIX A: STUDY METHODOLOGY 


Much of the commentary and information contained in this ! 

study was derived from 66 oral interviews with 97 individuals, 
many of whom have had extensive experience or direct acquain- 
tance with national intelligence estimates. Many, although IS 

not all, of the suggestions offered in the study stem directly I 

or indirectly from their views of what needs to be done to IS 

improve the product and the process. a 

Interviewees were contacted by members of the study team I 


principally because we knew them, or knew of them, and believed SI 

they would have some useful views on the subjects included in 
the study. Inevitably, we contacted many more individuals than ii: 

were actually interviewed. Some meetings we ardently hoped to M 

arrange simply proved impossible from the start, or had to be 1 

scrubbed at the last minute, because of the press of other 3 

obligations on some very busy people. In no sense, then, | 

can this be considered a "scientifically" constructed inter- fi 

view base. We did try very hard, however, to cover the whole |! 

spectrum of officials involved with estimates both on the i 

consumer and on the producer sides. We also tried to avoid ii 

giving undue weight or attention to the views of any one 
element or sector. 

It proved impossible to cover the entire range of ft 

issues addressed in this study with any one individual. i 
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While most of the people we talked with gave generously of 
their time, inevitably there was not enough opportunity with 
many of them to cover more than several of the many broad 
questions we wanted to take up. It did help that three and 
at times all four members of the study team attended the 
interviews, thereby enabling us to cover more ground and com- 
pare impressions afterwards of what was said and meant. 
Careful notes were taken. 

Consum ers 

We talked with 46 individuals whom we could classify as 
current users, to one degree or another, of estimative prod- 
ucts, and we interviewed seven prominent former consumers, 
whose reflections on estimates as they viewed them from some 
very senior positions promised to provide an additional 
dimension to the study. 

Of the current consumers, seven were National Security 
Council staff members or their deputies. We spoke with 21 
consumers in the Department of State, including one Under- 
secretary, the Counselor, 11 officers at the Assistant 
Secretary or Deputy Assistant Secretary level, six at the 
level of country director, deputy country director, or desk 
officer, and two officers of the Policy Planning Staff. At 
the Pentagon we interviewed 12 consumers, including eight 
in the International Security Affairs office ranging from 
the Deputy Assistant Secretary level on down, two in the 
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Office of the Secretary of Defense, and two general officers 
on the staff of the Joint Chiefs. We also talked with four 
officials in the economic policy realm- -three in the Department 
of the Treasury, and the Special Representative for Inter- 
national Trade Negotiations. We interviewed one officer of 
the Arms Control and Disarmament Agency and the Executive 
Secretary of the President's Foreign Intelligence Advisory 
Board. 

The former consumers whom we interviewed once held the 
following posts: Secretary of State, Secretary of Defense 
(two individuals), Assistant to the President for National 
Security Affairs, Assistant Secretary of Defense for Inter- 
national Security Affairs, Director of the Arms Control and 
Disarmament Agency, and Executive Secretary of the National 
Security Council. All had insightful and valuable comments 
and suggestions to offer. 

Producers 

We Interviewed 39 individuals whom we classified as 
intelligence producers with current responsibilities relating 
to some aspect of the estimative production process. We also 
talked with five former members of the Board of National 
Estimates who are now retired. 

Current producers whom we talked to in considerable 
detail included ten National Intelligence Officers or their 
deputies. Non-CIA intelligence producers we saw included 
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the Director and the Deputy Director (for Estimates) of the 
Defense Intelligence Agency, and the Director and Deputy 
Director of the Department of State's Bureau of Intelligence 
and Research. 

We interviewed or held seminars with 26 officers of 
CIA's Directorate of Intelligence, including the Deputy 
Director for Intelligence, five analysts and managers from 
the former Office of Current Intelligence, six from the 
former Office of Political Research, six from the Office of 
Strategic Research, six from the Office of Economic Research, 
and one each from the Office of Scientific Intelligence and 
the Office of Weapons Intelligence. 

Use of Estimates and Other Writings 

From the outset, it proved difficult to develop a workable 
plan for using national intelligence estimates themselves in 
the study. Because the existing body of estimative literature 
includes so many different kinds of products published on such 
a wide variety of subjects over the past 27 years, the prob- 
lem of choosing a representative sample to dissect and comment 
upon would have been formidable enough by itself. Assuming 
that such a sample could have been assembled, the study team 

i 

would have been severely limited in its ability to comment 
substantively on all but the relatively small number dealing 
with topics on which team members had some degree of personal 
expertise. Exploring in detail the processes by which all 
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but the most recent estimates have been produced would have 
been almost as difficult; few records are available, and 
personal recollections of participants usually fade rapidly 
and become untrustworthy. 

We also derived relatively little help from earlier 
studies and commentaries on the estimative record per se. 

No comprehensive look at estimates as a whole has ever been 
undertaken, and only a handful of post-mortems or validity 
studies on series of estimates have been done, although those 
that were written were consulted and proved useful. Reading 
those studies helped to reaffirm to us the wisdom of avoiding 
comment on the accuracy of specific estimates. This is 
devilishly hard to do with any real measure of validity. 

Many estimative judgments are conditional, i.e., their 
accuracy hinges on something else happening that may or 
may not occur. Others will prove "wrong" simply because 
the United States or some third country will have taken 
steps that altered the situation that obtained when the 
judgments were made. Of those that can more readily be 
identified years later as "right" or "wrong" calls, some 
will have been of major and some of minor importance, and 
it would be difficult to weigh them accurately now. Finally, 
to be able to conclude whether a past estimate was either on 
or off the mark, one would have to know his history well, or 
do a lot of research to find out what really did happen. 
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For these reasons, we decided to limit our use oi the 
texts of estimates to the study of some of those issued over 
the past year or two and to the identification of specific 
problems or issues concerning them by seeking comments and 
feedback from users. In the event, most of the comments we 
gleaned were too general or fragmentary to give us more than 
a few leads for further investigation. 

A fairly extensive although fragmentary body of literature 
about estimates and their role in the policy-making process 
does exist and was used. Especially helpful were the more 
than three dozen articles on various aspects of the estimative 
business that have been included in the quarterly publication, 
Studies in Intelligence , since 1957. The reports of the Com- 
mission on the Organization of the Government for the Conduct 
of Foreign Policy (the Murphy Commission) and of the Senate 
Select Committee on Intelligence were very useful. Various 
internal CIA memoranda, histories, and reports on estimates 
also proved helpful, as did selected portions of books, 
magazine articles, and other open literature concerning 
estimates and their role in the policy-making process. 

Most of our data, however, came from the people we interviewed. 
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APPENDIX B: SELECTIVE BIBLIOGRAPHY 


I. Material in the following unclassified publications that 

were among those used by members of the study team appeared 
especially relevant to issues discussed in the study. 

A. Books 
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Hilsman, Roger, The Politics o f Policy Making in 
Defense and For eign Affairs, (Harper 5 Row, 

New York, 1971) . 

_ > To Move a Nation, (Dell. New York. 

1974). ' 

Kent, Sherman, Strategic Inte lligence for American 

World Policy , (Princeton University Press, 1949). 

Kirkpatrick, Lyman B., The U.S. Intelligence 
Community , (Hill § Wang, New York, 1973) . 

Moorstein, Richard, and Abramowitz, Morton, 

Remaking China Policy , (Harvard University 
Press , 1971) . 

deRivera, Joseph H. , The Psychological Dimension 
of Foreign Policy , (Merrill Co., Columbus, 

Ohio, 1968). 

Wohlstetter, Roberta, Pearl Harbor--Warning and 
Decision , (Stanford University Press, 1962). 

B. Reports 


Final Report, Senate Select Committee to Study 
Governmental Operations with Respect to 
Intelligence Activities, (U.S. Government 
Printing Office, Washington, D.C., 1976). 

Final Report, Commission on the Organization of 
the Government for the Conduct of Foreign 
Policy, (U.S. Government Printing Office, 
Washington, D.C., 1975), Volume 7. 
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C . Magazine and Journal Articles 

Ben- Zvi , Abraham, "Hindsight and Foresight: A 
Conceptual Framework for the Analysis of 
Surprise Attacks," World Politics . 

Cline, Ray S., "Policy Without Intelligence," 
Foreign Policy , Winter 1974-75. 

Cooper, Chester L., "The CIA and Decision-Making," 
Foreign Affairs , January 1972 . 

Graham, MG Daniel 0., "Estimating the Threat: A 
Soldier's Job," Army Ma gazine , April 1973. 

Hughes, Thomas L., "The Power to Speak and the 
Power to Listen," in Secrecy and Foreign 
Policy, Thomas M. Franck § Edward “Wei sband, 
eds., (Oxford University Press, 1974). 

Kent, Sherman, "Estimates and Influence," Foreign 
Service Journal , April 1969. 

Knorr, Klaus, "Failures in NIEs : The Case of the 
Cuban Missiles," World Politics , April 1964. 

Shlaim, Avi, "Failures in NIEs: The Case of the 
Yom Kippur War," World Politics , April 1975. 

Wohlstetter , Albert, "Is There a Strategic Arms 
Race?" Foreign Policy , Summer 1974. 

_, "Rivals But No Race," Foreign 

Policy , Fall 1974. 


, "Optimal Ways to Confuse Ourselves," 

Foreign Policy , Fall 1975. 

II. The following classified reports, memoranda, articles, and 
studies were found to be especially pertinent to some of 
the issues taken up in this study. 

A. Articles in Studies in Intelligence (all issues 
classified Secret) . 
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B. Miscellaneous Official Studies, Memoranda, and Reports 


Center for the Study of Intelligence, Intelligence 
Monograph, CIA Intelligence Support for Foreign 
and National Security Pol icy ~" Making , (tr/im 76-01 
January 19 76 , S) . 


, Intelligence 

Monograph, The Future Market for Finished 
Intelligence , (TR/IM 76-02, August 1976, S) . 

Intelligence Community Staff, An Assessment of N ational 
F oreign Intelligence Support to the U. S, Governmen t , 
^December 1976, S) . 

, Analysis of Intelligence 

bn the Middle East War - October 1973 , (August 
1974, TS-Sensitive) . 

Kent, Sherman, The Law and Custom of the National 
Intelligence Estim ate, (February 1976, S - CIA 
Internal Use Only) . 

Office of National Estimates, Memorandum, 1967 1 s 
Estimative Record--Five Years Later , ( C I A/ ONE 
August 1972 , TS) . 


, Memorandum for the DCI, 

NIEs on the Soviet Role in the Middle East, 1965- 
1970 , (CIA/ ONE July 1970, S-Sensitive) 

, Study, NIEs on the USSR , 
1950-1957 , (CIA/ONE May 1958, TS) . 

, Staff Memorandum, The 

Arabs and the Estimates, 1951-1965 , (CIA/ ONE 
November 1965, TS) . 

Office of Research and Development, Report o n Proceeding s, 
Symposium on Analytical Methodology, (CIA/ORD , 

Report No. RD/CDAM- 1 , June 1974",' Cj. 
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